RE: action item 3913 - Concept for binary data format

["Hubert Heijkers" <Hubert.Heijkers@nl.ibm.com>]

Hi Christof,

Yeah, that doc is what is supposed to be the start of the formal normative specification for Compact JSON, if you haven't seen my contribution from back then you can find it here: https://www.oasis-open.org/apps/org/workgroup/odata/download.php/62898/odata-compact-json-format-v4.0.pdf

This was enthusiastically received by and I had a follow up meeting with the SAP guys, at their offices, a week or two later, which lead to a number of great ideas/improvements. I'm still eager to take that next step and then move to a binary 'equivalent'.

Let me know what you think, happy to discuss further.

Cheers,

-H


Hubert Heijkers | STSM, Chief Architect TM1 Server and OData Evangelist | IBM Business Analytics | +31-621-394123
"The crisis of today is the joke of tomorrow" - H.G. Wells

Christof Sprenger ---03-08-2020 05:23:33 PM---Thanks Hubert for the reply. Would you please share the current stage of the discussion on compact J

From: Christof Sprenger <chrispre@microsoft.com>
To: Hubert Heijkers <Hubert.Heijkers@nl.ibm.com>
Cc: "odata@lists.oasis-open.org" <odata@lists.oasis-open.org>
Date: 03-08-2020 05:23 PM
Subject: RE: [EXTERNAL] Re: action item 3913 - Concept for binary data format

---

Thanks Hubert for the reply.
 
Would you please share the current stage of the discussion on compact JSON ? I was only able to find this document which is relatively empty.
 
There is a fundamental difference between MessagePack and ProtoBuf. In ProtoBuf message format is defined upfront and needs to be known and carefully versioned for compatibility. MessagePack, has a fixed but very flexible and simple model (see type system , and yes, arrayâs are length prefixed similar to all variable length structures like maps, strings, â). In that sense this is similar to JSON and XML where typically there is a second step in Memory that transforms message into Programming Language Types. 
In summary, my impression is that MessagePack is so much closer to JSON that I would expect that it should be easier for any OData implementation to add such a format.
 
As mentioned above, I do not have any insights into the discussion on compact JSON but it might also be worth investigating compression on top of a binary format.
 
Happy to help with any investigations.
 
Christof
 
From: Hubert Heijkers <Hubert.Heijkers@nl.ibm.com> 
Sent: Sunday, August 2, 2020 09:19
To: Christof Sprenger <chrispre@microsoft.com>
Cc: odata@lists.oasis-open.org
Subject: [EXTERNAL] Re: action item 3913 - Concept for binary data format

I still have a note here to reach back out to Mike as well because he mentioned ProtoBuf a couple of months ago by now as well in this same realm.

For me this has always been an action item that was closely related to the Compact JSON work, hoping to iron out how we'd be able to specify very concisely what's to be expected in the payload in Compact JSON already and then come up with a representation in which we could use a more binary (read: less human readable) format to optimize the communication further. Back then I'd looked at a bunch of binary formats that would potentially make sense, MessagePack and ProtoBuf (the later one even more so since we use build on GRPC), but the time couldn't find any that I liked because either 1) they didn't have a separation between metadata data and a data file, 2) didn't support collections without needing to know the size up front easily (a definite no go for us) or 3) didn't feel we could easily be embedded in our serialization abstraction logic.

So to answer your question, other then early investigations, I've not gotten to this again, wasn't planning to until we'd have taken the next step on Compact JSON (which is effectively taking my proposal - which we have in production - and merge in all the ideas/thoughts we had since and come up with a normative proposal for it).

Cheers,

-H


Hubert Heijkers | STSM, Chief Architect TM1 Server and OData Evangelist | IBM Business Analytics | +31-621-394123
"The crisis of today is the joke of tomorrow" - H.G. Wells

Christof Sprenger ---31-07-2020 08:29:54 PM---Hubert, We recently stumbled upon MessagePack<INVALID URI REMOVED

From: Christof Sprenger <chrispre@microsoft.com>
To: Hubert Heijkers <hubert.heijkers@nl.ibm.com>
Cc: "odata@lists.oasis-open.org" <odata@lists.oasis-open.org>
Date: 31-07-2020 08:29 PM
Subject: [EXTERNAL] action item 3913 - Concept for binary data format

---

Hubert,

We recently stumbled upon MessagePack and I remembered the action item 3913 which seems to be assigned to you.
May I ask in which stage this action item is? I would love to explore if/how Message Pack fits into this since it appears to be at first sight so similar to JSON that this should give a nice balance between efficiency and compatibility.

Christof





Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214

Tenzij hierboven anders aangegeven: / Unless stated otherwise above:
IBM Nederland B.V.
Gevestigd te Amsterdam
Inschrijving Handelsregister Amsterdam Nr. 33054214