[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] New Work - Pagination
Jason,
Most of these issues are already resolved in TAXII 2. Let me explain.
1) Time synchronization is really not an issue. Given the header fields we added to TAXII 2, the client does not need to really know nor does it care what time the server is using. Also the time sync in TAXII 2 is not the same as TAXII 1.
2) The time zone aspect also is not an issue as the data is all tied to the date / time on the server.
3) The number of records is limited by the server. What we do not have right now is a way for the client to tell the server that it only wants 50 records not the 1000 that the server limits it to .
So for example...
The client will make a request to a server. The server will respond back with the number of records it allows and it will have two header fields that give the time book marks for the data as it was add to the server. This is NOT the STIX timeframes (which was the problem in TAXII 1).
The client can then use that information to make additional calls. Right now we only offer an added_after filter parameter. So the client can take the last time stamp given in the header and use that to make the next request. What the client can not currently do is walk backwards. This is why I think we should add an added_before filter parameter.
Does this help explain things? I think our time based sync can do greater than 80% of the use cases, especially if we added two filter parameters: a) limit (the client limit of the number of records it wants) b) added_before (the ability for the client to walk backwards from a given time
Bret
From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent: Monday, September 17, 2018 10:57:01 AM To: Bret Jordan Cc: cti-taxii@lists.oasis-open.org Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] New Work - Pagination The problems with date based pagination are well documented from the TAXII 1 days. In addition to the issue below, you have
* Time synchronization. Quite often the two systems won't have their clocks synced, nor any way to do so since one is owned by a third party. * You also then have them running in different time zones and have their hardware clocks set improperly instead of just running them on UTC. * Finally you have the problem where you are unsure what date range you should pass, so that you get an amount of records you can handle, since on 1 hour of data might contain 100 records, and the next hour might contain 1,000,000 Those reasons (and some others) are actually what led us to adding item-based pagination as an option. - Jason Keirstead Lead Architect - IBM.Security www.ibm.com/security "Things may come to those who wait, but only the things left by those who hustle." - Unknown From: Bret Jordan <Bret_Jordan@symantec.com> To: Jason Keirstead <Jason.Keirstead@ca.ibm.com> Cc: "cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org> Date: 09/17/2018 01:50 PM Subject: [cti-taxii] Re: [EXT] Re: [cti-taxii] New Work - Pagination Sent by: <cti-taxii@lists.oasis-open.org> I am not suggesting that we change the filtering by added_after at all. I personally believe you are correct in thinking that one could paginate by added_after in most circumstances. I also think an added_before would be a nice addition. The only problem I have heard about pagination in regards to using a date added value is you could run into a problem if large amounts of records were added in a single transaction and that transaction used the same "date added" for every record in the transaction. One way some systems might address this is to use microsecond precision for the date added value and then try and ensure each record gets its own timestamp or that no more than X records use the same microsecond timestamp. This should give enough room, even under heavy load. But I am sure as soon as I say this, people will have an example where that will even fall over. Bret From: Jason Keirstead <Jason.Keirstead@ca.ibm.com> Sent: Monday, September 17, 2018 10:29:52 AM To: Bret Jordan Cc: cti-taxii@lists.oasis-open.org Subject: [EXT] Re: [cti-taxii] New Work - Pagination Can we outline the problem sets / challenges with item-based paging, so everyone is on the same page WRT the discussion? As well, currently you can already "page by date" in TAXII (like you could in TAXII 1) - so in effect we have two mechanisms already. Are we talking about changing that as well? - Jason Keirstead Lead Architect - IBM.Security www.ibm.com/security "Things may come to those who wait, but only the things left by those who hustle." - Unknown From: Bret Jordan <Bret_Jordan@symantec.com> To: "cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org> Date: 09/17/2018 01:11 PM Subject: [cti-taxii] New Work - Pagination Sent by: <cti-taxii@lists.oasis-open.org> All, Now that TAXII 2.1 Working Draft 03 is up for Committee Specification Draft 01 ballot, I would like to start requesting proposals for how we should address and solve pagination for TAXII 2.1. As you know, we tried to use HTTP's native "Item" based pagination in TAXII 2.0, but after doing additional research and trying to implement this, we discovered that not only is this a known "do not do for RESTful APIs" but it is also near impossible to implement in code given the STIX dataset. I know several of you have expressed ideas for how this could be done/fixed in TAXII 2.1, so I am asking as one of your sub-committee co-chairs to please write up your ideas and submit them to the TAXII email list as proposals. Thanks Bret |
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]