[OASIS Issue Tracker] (ODATA-868) Describe HTTP encoding for streamed requests and responses

[OASIS Issues Tracker <workgroup_mailer@lists.oasis-open.org>]

    [ https://issues.oasis-open.org/browse/ODATA-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=62831#comment-62831 ] 

Hubert Heijkers commented on ODATA-868:
---------------------------------------

To be honest I've only thought of streaming in a sense of ordering things in the response so that consumers do not need to parse the complete JSON object (read: response) to be able to interpret it. What you seem to be questioning is the response couldn't effectively be a stream of smaller responses that each would be 'valid' JSON objects (or collections).

I've not looked at HTTP/2 much, and when I did it wasn't specific for OData - albeit for putting a more 'efficient' layer on top of my otherwise OData compliant API, but I haven't gotten the impression it necessarily supports something out of the box for this. It supports streams, and the big thing being - I think - the fact you can have more then one in parallel on one connection or even multiple connections(?), but I don't think it says anything about what's in those "chunks" that you put on the stream that make up the message that's being send over that stream correct?

Even with HTTP/1 encoding one doesn't necessarily need chunked transfer encoding to pass along a big response albeit that you likely have less control over it that way, plus you need to know what the total size is before you start sending it. Streaming with chunked transfer encoding, at least in my case, is interesting because it allows the server to start composing/sending the response without even knowing how much is yet to follow (if anything). Obviously my service always adheres to the ordering as defined in our specification allowing the consumer to start parsing the, yes incomplete, response if he/she is set up to do so to begin with.
From the service side, inspired by a comment from Mike years ago, we simply buffer up to a max chunk size, something like 64kb and then send a chunk. If an error occurs before that buffer limit is reached then an error is returned and nothing that was written to the buffer was ever send. If the whole response fits in that buffer chunking never gets used and the response is send with a response size header and everything. If however an error occurs after at least one chunk has been send then the HTTP connection simply gets terminated. Because the final empty chunk wasn't sent that gets treated by all clients that we are aware of as an error, however it doesn't allow the service to relay the actual error obviously.

I understand you idea based on the server driven paging, although my service doesn't do that (client always gets what he asks for), but that would technically make the response, irrespective of which HTTP version as per my understanding, invalid so in that case we would need a very explicit content-type, or at least a content-type parameter of which the client expressed he supports that, to relay that when using chunked transfer encoding that each chunk by itself contains a JSON object in our case.
I also suppose you are only envision this then to be only used in apps that really know that they are dealing with an OData service right and not by a generic one like a browser correct? A browser would merge all the chunks and, again based on content-type, interpret as one big JSON response if we are not going to use an OData specific content-type in this case.

I can definitely see benefits of, optionally, adding notion of streaming to the one/single response but, if we do, we have to make sure we don't trip any oblivious consumers.

> Describe HTTP encoding for streamed requests and responses
> ----------------------------------------------------------
>
>                 Key: ODATA-868
>                 URL: https://issues.oasis-open.org/browse/ODATA-868
>             Project: OASIS Open Data Protocol (OData) TC
>          Issue Type: Bug
>          Components: OData JSON Format
>    Affects Versions: V4.0_ERRATA02
>            Reporter: Ralf Handl
>              Labels: Clarification
>             Fix For: V4.01_WD01
>
>
> OData JSON Format has a format parameter for streaming scenarios but does not describe how these messages should be encoded.
> Part 1: Protocol, section 9.4 talks about in-stream errors in a format-independent way. Unfortunately we don't define how these should be represented in the JSON format.
> Situations where in-stream errors can occur include
> - metadata requests in XML or JSON
> - data request in JSON or XML
> - media resource requests with any content-type



--
This message was sent by Atlassian JIRA
(v6.2.2#6258)