Re: [xdi] EUREKA!!! (Or, in current lingo, OMG!!!) New XDI JSONserialization format

[Drummond Reed <drummond.reed@xdi.org>]

Thanks, Mike, I'm very psyched about it too. I couldn't stop thinking about it all weekend.

If others agree, the next step is to update the JSON Format page at http://wiki.oasis-open.org/xdi/JsonFormat.

=Drummond

On Sat, Mar 26, 2011 at 7:21 AM, Michael Schwartz <mike@gluu.org> wrote:

I am in favor of this format… Nice work Drummond.

 

-          Mike

 

 

 

From: drummond.reed@gmail.com [mailto:drummond.reed@gmail.com] On Behalf Of Drummond Reed
Sent: Friday, March 25, 2011 5:26 PM
To: OASIS - XDI TC
Subject: [xdi] EUREKA!!! (Or, in current lingo, OMG!!!) New XDI JSON serialization format

 

Although I am in a car traveling east of the Cascade Mountains in Washington, my wife is driving so I could compose this message before the weekend because I am so excited to share this breakthrough with the TC.

At the very end of yesterday's telecon, we noted that writing up the collection of XDI statements that correspond to each of the example graphs in the XDI Graph Patterns document, both Bill and I were struck by how compact and readable a representation of the graph they represented. And the fact that there is exactly one XDI statement for every arc in the graph is very clean and clear.

Our conclusion on yesterday's call was that this meant there may be a very simple serialization of the graph based directly on the XDI statements.

Last night, to prepare for this morning's OpenXDI telecon, I wrote up the collection of XDI statements for the second example graph ("Simple Properties"), and I started thinking about how this collection would be serialized in JSON. It didn't seem very hard, but I was too tired to work it through.

This morning when I woke up, as soon as my mind returned to the problem, suddenly the solution was obvious: there was a super-simple way to serialize the graph using pure graph statements AND KEEP ALL THE ADVANTAGES OF OUR JSON SERIALIZATION FORMAT.

In other words, there is a much simpler JSON serialization that still supports using all the same native JSON datatypes.

Here are the rules:

1. The entire graph is serialized as one top-level JSON object containing one-or-more JSON objects.
2. Every contained object consists of a key and a value.
3. The key MUST be the first two segments of the XDI statement representing the XDI subject and XDI predicate.
4. The value MUST be an array.
5. If the key ends in the character !, then the value in the array is an XDI literal.
6. If the key ends in any other character, the value(s) in the array are XRIs.

The advantages of this format:

1. It is very simple and flat -- no JSON nesting is needed because all the nesting is reflected in the structure of the XRIs used to create the keys.
2. It is completely indexed, i.e., code can navigate directly to any value in the entire serialized graph just by looking up the key.
3. The JSON datatype returned for all objects is the same (an array), making it easy for code to process. (Mike likes this.)
4. All native JSON datatypes -- of arbitrary complexity -- are fully supported as literal values inside this array.
5. This format is so simple that it will allow us to do away with needing any other format (i.e., X2, X3) for human-readable/writeable serialization of the graph.

In retrospect, there are two secrets behind this dramatically simpler serialization format, neither of which was apparent at Whistler last August:

1. Although we had the seed of the idea at Whistler that we could use an XRI delimiter to signal that an XDI object was a literal, we didn't have the right semantics for what that delimiter was. Now that we have a trailing delimiter (!) for indicating a literal, it is easy.
2. This format uses both the first and second segments of each XDI statement as the key. All of our earlier formats had focused on just using the first segment as a key. Although using the first and second segments creates a small amount of redundant XRI usage in the serialization format, this is strongly offset by the simplicity and speed of processing. (The irony is that using a 2-segment key/value structure for serializing the graph was originally worked out and implemented my Markus about a year ago.)

Below are examples of both the collection of XDI statements and this new JSON serialization of the "Primary and Secondary Addressing" and "Simple Property" example graphs in the XDI Graph Patterns document. (Note that they don't yet reflect a few of the changes discussed on yesterday's and today's calls.)

Please do reply to this message with your thoughts about whether this should become our new official JSON serialization format. If so, I will post a format definition and start updating the XDI Graph Model document.

Have a great weekend,

=Drummond

Primary and Secondary Addresses

XDI Statements

()/$$/(=!0999.a7b2.25fd.c609)

()/$$/(http://example.com/ox/=!0999.a7b2.25fd.c609)

()/()/(=!0999.a7b2.25fd.c609)

()/()/(http://example.com/ox/=!0999.a7b2.25fd.c609)

()/()/=abc

()/()/=!0999.a7b2.25fd.c609

=abc/$/=!0999.a7b2.25fd.c609

=!0999.a7b2.25fd.c609/$$/=abc

=!0999.a7b2.25fd.c609/()/+pea-patch

=!0999.a7b2.25fd.c609/()/+garden

=!0999.a7b2.25fd.c609/()/!1

=!0999.a7b2.25fd.c609+pea-patch/$/=!0999.a7b2.25fd.c609!1

=!0999.a7b2.25fd.c609+garden/$/=!0999.a7b2.25fd.c609!1

Proposed New JSON Serialization

{

            "()/$$": [

                        "(=!0999.a7b2.25fd.c609)",

                        " (http://example.com/ox/=!0999.a7b2.25fd.c609)"

            ],

            "()/()": [

                        "(=!0999.a7b2.25fd.c609)",

                        "(http://example.com/ox/=!0999.a7b2.25fd.c609)",

                        "=abc",

                        "=!0999.a7b2.25fd.c609"

            ],

            "=abc/$": [

                        "=!0999.a7b2.25fd.c609"

            ],

            "=!0999.a7b2.25fd.c609/$$": [

                        "=abc"

            ],

            "=!0999.a7b2.25fd.c609/()": [

                        "+pea-patch",

                        "+garden"

            ],

            "=!0999.a7b2.25fd.c609/()": [

                        "!1"

            ],

            "=!0999.a7b2.25fd.c609+pea-patch/$": [

                        "=!0999.a7b2.25fd.c609!1"

            ],

            "=!0999.a7b2.25fd.c609+garden/$": [

                        "=!0999.a7b2.25fd.c609!1"

            ],

}

Simple Properties

XDI Statements

()/()/=abc

=abc/+age!/(data:33)

=abc/()/+age

=abc+age/$d!/(data:2010-10-10T11:12:13Z)

=abc+age/()/$v

=abc+age$v/!1!/(data:32)

=abc+age$v/!2!/(=abc/+age!)

=abc+age$v/()/!1

=abc+age$v/()/!2

=abc+age$v!1/$d!/(data:2010-09-09T10:11:12Z)

=abc+age$v!2/$d!/(data:2010-10-10T11:12:13Z)

Proposed New JSON Serialization

{

            "()/()": [

                        "=abc"

            ],

            "=abc/+age!": [

                        33

            ],

            "=abc/()": [

                        "+age"

            ],

            "=abc+age/$d!": [

                        "2010-10-10T11:12:13Z"

            ],

            "=abc+age/()": [

                        "$v"

            ],

            "=abc+age$v/!1!": [

                        32

            ],

            "=abc+age$v/!2!": [

                        "(=abc/+age!)"

            ],

            "=abc+age$v/()": [

                        "!1",

                        "!2"

            ],

            "=abc+age$v!1/$d!": [

                        "2010-09-09T10:11:12Z"

            ],

            "=abc+age$v!2/$d!": [

                        "2010-10-10T11:12:13Z"

            ]

}