Re: [cti] TAXII 2.1 WD 09 reminder

[Matt Pladna <mpladna@lookingglasscyber.com>]

Bret,

 

Thanks for the reminder.  I reviewed the pagination text in WD 09.  I donât believe itâs clear enough about the two pagination methods being allowed: Range/Content-Range headers from 2.0 and Limit Parameter in 2.1.  I took a stab at drafting up some text below.  Note that my headers are probably not sized right for be sub sections.

 

Iâd appreciate you and a couple others reviewing and weighing in.  Iâve modified slightly the main pagination section and added subsections to clearly split the range pagination from limit param.

3.5 Pagination

TAXII 2.1 supports pagination of large result sets on certain endpoints. These endpoints return results sorted in ascending order by the date they were added to the collection (see section 3.3). The server may limit the number of responses in response to a query in one of two ways:

 

1. The server can use HTTP's Range and Content-Range (Range/Content-Range Pagination) headers, and defines the items range unit, to perform pagination as defined in section 2 of [RFC7233]. items is the mandatory to implement range unit for TAXII.  Other range units MAY be implemented by clients and servers.
2. A limit parameter (Limit parameter Pagination) can be used by a client for a request to a paginated endpoint (see section 3.4). 

 

If more objects are available after a paginated response, either because the client requested that they be limited via the limit parameter or the client/server limits them via Range and Content-Range headers, then the response envelope MUST have a value of true in the more property and MAY have an appropriate value in the next property.

 

If the more property is set to true and the next property is populated then the client can paginate through the remaining records using the next URL parameter along with the same original query options. 

 

If the more property is set to true and the next property is empty then the client may paginate through the remaining records by 

1. using the added_after URL parameter with the date/time value from the X-TAXII-Date-Added-Last header along with the same original query options.
2. Using Range and Content-Range headers

 

It is possible for the server to return "more": true in a response, yet present no additional objects in the follow-on query. This can occur when the additional objects are deleted from a collection between requests. 

3.5.1 Range/Content-Range Pagination

The items range unit is defined for expressing subranges of a resource [HTTP 7233]. For the Endpoints that return object, items represents objects. For the Endpoints that return collections, items represents Collections.

 

The first items value in the Range and Content-Range headers gives the first item in a range. The last items value in the Range and Content-Range headers gives the last item in the range; that is, items ranges specified are inclusive. items are zero-indexed; that is, the first item is object zero. A Content-Range header will have a third value that identifies the size of the available dataset. 

 

For example:

• If the Range header contains "items 10-49", "10" represents the first item requested; and "49" represents the last item requested.
• if the Content-Range header contains "items 10-49/500", "10" represents the first object in the response; "49" represents the last object in the response; and "500" represents the total number of items available.

 

All items values MUST be:

• a non-negative integer
• zero indexed (i.e., the first object is object "0")

 

â3.5.1.1â Range/Content-Range Requirements

The following requirements only apply to items based range requests

 

• The Accept-Ranges header allows a server to indicate that it supports range requests for the target resource [RFC7233] as well as which range units are supported.
• For resources where items-based pagination is supported, and where the Accept-Ranges header is present, the Accept-Ranges header MUST contain items as an acceptable range. The Accept-Ranges header MAY contain other acceptable ranges, if the server supports them. 
• Requests MAY use the Range header to request a subset of data that would otherwise be returned. 
• As defined in the HTTP specification, HTTP 206 (Partial Content) [RFC7233] responses include a Content-Range header, even if the entire resource is contained in the response.
• As defined in the HTTP specification, if the requested Range cannot be satisfied, an HTTP 416 (Requested Range Not Satisfiable) [RFC7233] response is used.
• For example, if a range requests items 500-600, but only 0-100 are available, an HTTP 416 (Requested Range Not Satisfiable) is used.
• An HTTP 206 (Partial Content) response with a Content-Range header MAY be returned even if the original request did not have a Range header.
• Note that this is a deviation from the HTTP specification, which indicates that HTTP 206 responses are only permitted when the Range header is present in the request.
• Responses to requests with a Range header SHOULD contain only the requested range and MAY include a range smaller than what was requested. 
• TAXII follows standard HTTP rules for the Content-Range and Range headers, with the exception of allowing a 206 response to a request without a Range header:
• The 206 (Partial Content) status code indicates that the server is successfully fulfilling a range request for the target resource see section 4.1 of [RFC7233]
• If a single part is being transferred, the server generating the 206 response MUST generate a Content-Range header field, describing what range of the selected representation is enclosed, and a payload consisting of the range. See section 4.2 of [RFC7233].

 

NOTE: The total number of items available in a result may change with each request for a new page in the paginated result set. This can happen if items have been added or deleted between subsequent requests.

3.5.2 Limit Parameter Pagination

• Collection High-Value-Indicators has 1000 records in it.
• The client or server has limited all responses to 100 records at a time.
• A client will make a request and the server will respond with the first 100 records.
• The server will also populate the two X headers for TAXII, X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last. These headers will contain the date/time value of when the first and last records of the 100 returned records were added to the TAXII server.
• The server will also set the more property to a value of true and may set the next property to an appropriate value on the TAXII envelope.
• When a client wants to obtain the next 100 records, the client will do either:
• Provided the same query/filter parameters that it originally used along with populating the next URL parameter with the value from the next property on the previous TAXII envelope response.
• Or provide the same query/filter parameters that it originally used along with the date/time value from the X-TAXII-Date-Added-Last in the added_after URL parameter 

 

 

From: <cti@lists.oasis-open.org> on behalf of Bret Jordan <Bret_Jordan@symantec.com>
Date: Friday, October 25, 2019 at 13:31
To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: [cti] TAXII 2.1 WD 09 reminder

 

All,

 

Just a reminder that all comments and suggestions for TAXII 2.1 WD09 are due today. 

 

Thanks

Bret