OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

cti-taxii message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] New Work - Pagination


Jason,


Most of these issues are already resolved in TAXII 2.  Let me explain.


1) Time synchronization is really not an issue. Given the header fields we added to TAXII 2, the client does not need to really know nor does it care what time the server is using. Also the time sync in TAXII 2 is not the same as TAXII 1.


2) The time zone aspect also is not an issue as the data is all tied to the date / time on the server.


3) The number of records is limited by the server.  What we do not have right now is a way for the client to tell the server that it only wants 50 records not the 1000 that the server limits it to .


So for example...


The client will make a request to a server.  The server will respond back with the number of records it allows and it will have two header fields that give the time book marks for the data as it was add to the server.  This is NOT the STIX timeframes (which was the problem in TAXII 1). 


The client can then use that information to make additional calls.  Right now we only offer an added_after filter parameter.  So the client can take the last time stamp given in the header and use that to make the next request. What the client can not currently do is walk backwards.  This is why I think we should add an added_before filter parameter.


Does this help explain things?  I think our time based sync can do greater than 80% of the use cases, especially if we added two filter parameters:

a) limit (the client limit of the number of records it wants)

b) added_before (the ability for the client to walk backwards from a given time 


Bret



From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent: Monday, September 17, 2018 10:57:01 AM
To: Bret Jordan
Cc: cti-taxii@lists.oasis-open.org
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] New Work - Pagination
 
The problems with date based pagination are well documented from the TAXII 1 days. In addition to the issue below, you have

* Time synchronization. Quite often the two systems won't have their clocks synced, nor any way to do so since one is owned by a third party.

* You also then have them running in different time zones and have their hardware clocks set improperly instead of just running them on UTC.

* Finally you have the problem where you are unsure what date range you should pass, so that you get an amount of records you can handle, since on 1 hour of data might contain 100 records, and the next hour might contain 1,000,000

Those reasons (and some others) are actually what led us to adding item-based pagination as an option.
-
Jason Keirstead
Lead Architect - IBM.Security
www.ibm.com/security

"Things may come to those who wait, but only the things left by those who hustle." - Unknown




From:        Bret Jordan <Bret_Jordan@symantec.com>
To:        Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc:        "cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Date:        09/17/2018 01:50 PM
Subject:        [cti-taxii] Re: [EXT] Re: [cti-taxii] New Work - Pagination
Sent by:        <cti-taxii@lists.oasis-open.org>




I am not suggesting that we change the filtering by added_after at all.  I personally believe you are correct in thinking that one could paginate by added_after in most circumstances.  I also think an added_before would be a nice addition.  

The only problem I have heard about pagination in regards to using a date added value is you could run into a problem if large amounts of records were added in a single transaction and that transaction used the same "date added" for every record in the transaction.

One way some systems might address this is to use microsecond precision for the date added value and then try and ensure each record gets its own timestamp or that no more than X records use the same microsecond timestamp.  This should give enough room, even under heavy load.  But I am sure as soon as I say this, people will have an example where that will even fall over.

Bret



From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent:
Monday, September 17, 2018 10:29:52 AM
To:
Bret Jordan
Cc:
cti-taxii@lists.oasis-open.org
Subject:
[EXT] Re: [cti-taxii] New Work - Pagination

 
Can we outline the problem sets / challenges with item-based paging, so everyone is on the same page WRT the discussion?

As well, currently  you can already "page by date" in TAXII (like you could in TAXII 1) - so in effect we have two mechanisms already. Are we talking about changing that as well?


-
Jason Keirstead
Lead Architect - IBM.Security

www.ibm.com/security

"Things may come to those who wait, but only the things left by those who hustle." - Unknown





From:        
Bret Jordan <Bret_Jordan@symantec.com>
To:        
"cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Date:        
09/17/2018 01:11 PM
Subject:        
[cti-taxii] New Work - Pagination
Sent by:        
<cti-taxii@lists.oasis-open.org>




All,


Now that TAXII 2.1 Working Draft 03 is up for Committee Specification Draft 01 ballot, I would like to start requesting proposals for how we should address and solve pagination for TAXII 2.1.  As you know, we tried to use HTTP's native "Item" based pagination in TAXII 2.0, but after doing additional research and trying to implement this, we discovered that not only is this a known "do not do for RESTful APIs" but it is also near impossible to implement in code given the STIX dataset.


I know several of you have expressed ideas for how this could be done/fixed in TAXII 2.1, so I am asking as one of your sub-committee co-chairs to please write up your ideas and submit them to the TAXII email list as proposals.


Thanks
Bret









[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]