Re: [amqp] addressing use cases

[Rafael Schloming <rhs@datawire.io>]

On Tue, Feb 17, 2015 at 10:10 AM, John O'Hara <john@rjohara.com> wrote:

Hi Rafi

I hope you are well; it's be an while since I've written.

You've managed to coax me out of my relative silence :-)

Hi John, it's good to hear from you. ;-)

I can't quite get my head around this proposal without understanding the motivation and use cases.

What is the use case that REST does not sufficiently cover in this space?

Technology duplication effort might be better invested in tougher problems.

The directed flow of communications in AMQP is what makes it very powerful and very useful.  The (original) absence of needing hostnames and port numbers to address application level concerns was incredibly useful for large scale applications.  Anyone who has struggled to represent multiple deployment environments (e.g. for testing) will understand how painful host:port is.

SEDA - Staged Event Driven Architecture and

CQRS - Command Query Responsibility Segregation

are both incredibly powerful architectural patterns that AMQP makes super easy.

I like the idea of getting some conventions together for how to represent resources and imperatives (commands) and for patters to return data to streaming requests (to a different destination), I don't understand why we need to copy REST.  The main failing of REST is its tight temporal coupling - and this proposal seeks to create similar fragility in AMQP, if I understand correctly (which I may not).

Apologies if I'm getting this completely wrong.

No worries, I'll try to expand a bit. FWIW, I wouldn't read that much into the "REST-like" analogy. The like is in there for a reason. Also, this isn't actually a proposal, just describing some scenarios where giving applications built on top of amqp more flexibility and customization  around the definition of sources and targets would be quite useful. The current source and target definitions are rather specific to the JMS 1.0 flavor of queues and topics, and don't map super well to some less JMS-like queue and topic implementations, or indeed even to some of the additions made in JMS 2.0.

A good example here are kafka-style message stores. The basic model of a kafka-style message store is a theoretically infinite linear sequence of messages. The location of every subscriber is tracked by a single sequence-id that functions as a pointer into the message store. In a store like this, the server doesn't hold any per client subscription state for inactive clients. Instead a detached client is responsible for remembering its own pointer into the store and supplying it as a starting point when re-establishing/recovering a link. This differs from the JMS model where the server holds some sort of per client cursor or subscription queue that it has to remember even if a client isn't actively using a given link. In the JMS model, the client recovers this state by remembering an arbitrary client-id that correlates back to what the server is holding on behalf of the client.

Mapping a kafka-style message store into AMQP is difficult with the currently defined sources and targets as there is not an obvious place to map the "starting point". It is basically an application-level parameter to the source, and this is not a concept we currently have.

The reason I called this REST-like is in part that it is pushing what is traditionally server state around the subscription into the client, but as you can see it doesn't actually increase temporal coupling at all, in fact it significantly decreases coupling since you can recover a link without correlating through its initiating client.

Another scenario where flexibility around sources and targets would be useful is for realtime command and control situations. This is really much the same territory that the management spec is currently working through, except in a different domain. In my case I'm interested in supporting a scenario where I have a large number of devices deployed over a wide area. These devices produce time series data that ultimately gets fed through a processing/archiving pipeline. As you would expect, other than the scale, this is a pretty traditional non-realtime intermediated messaging scenario. The additional problem I have is that I also need the ability to connect directly to each device for realtime command and control operations. I have everything I need to establish AMQP links directly to these devices and send realtime messages back and forth. The only thing limiting me is that much like with the kafka-style store,  the AMQP definition of sources and targets don't have an obvious place for the application level parameters needed in my scenarios. I can stick these parameters in a variety of non-obvious places in the link handshake, but that feels a bit like making up my own standard which I'd rather avoid.

You might rightly point out that this latter sort of messaging is an example of the "tight temporal coupling" that you mention, but in this case that is the nature of the problem, not something imposed by the solution. I could of course deploy an entirely separate software stack onto my devices in order to handle that kind of communication, but hopefully you can see why I'm eager to avoid that. ;-)

These are a few more specific scenarios that I need to support. On the more abstract front, what I'd really like to enable is for applications to be able to naturally map their semantics directly into the primitives of AMQP (unidirectional message flows between sources and targets), the same way REST applications in the HTTP world map their semantics into the basic primitives of HTTP (GET, POST, etc). This is the other reason I make the analogy to REST. The key difference is that when you do this with HTTP, you are limited to the communication primitives that HTTP provides, i.e. just uploading and downloading files. When you do this with AMQP, you are building on a much more powerful and composable set of communication primitives, and your application can be quite trivially wired into a very flexible and powerful message network.

--Rafael