Here are some more details..
-
The concepts of items (what we have today) is only valid within a given instance of a single request. There is no guarantee that the server will have the same data associated with the same item numbers for a consecutive query.
- The server could add or delete data between queries and the client will either get redundant data or will miss data.
- About the best the server could do is hold a results table for a given TTL, and then expire it. So if the client does not make requests within that TTL, all bets are off. This solution would increase complexity and burden for no apparent value.
- A client will never be able to know the associated item numbers for data without the server. Meaning, if the client asked for 1,991,124,214 - 1,991,124,800, there is no predictability on what the client will get.
- About all the client will know is when was the last time it queried the collection and what did the server send as far as data.
So lets look at the use-cases of a client interacting with the server:
- Step 1: Has the client previously talked to the collection:
- If Yes: Then the client will know the timestamp of the query it last used and can use that information to gather information about what has changed.
- The client will make a request with an added_after url parameter and the server will respond with data up to the limit that the server will support. The client could provide a limit that it will accept. If it is greater than the server limit, then the
server limit wins. If the client limit is smaller than the server limit, then the client limit wins.
- The server will also send the ending timestamp for the data it is delivering. The client can then use this to get additional data.
- No: The client will just pull the collection and it will get the most current data. The server will also be able to tell the client the beginning timestamp and the ending timestamp for the data it sent and the server should be able to tell the client how
many total records there are in the collection.
- The client can use this information to gather additional data in the future by using the added_after URL parameter.
- The client can ask for additional historical data by using a added_before URL parameter.
This design will be a lot easier to implement and will not suffer the huge performance impacts of using item offsets.
I propose that we implement this solution, or a solution like this, and release a TAXII 2.1 CSD so that implementers can build their solutions around this design.
Bret
|