OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

amqp message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: AMQP Event Streams draft PR

Updated the PR per discussion on Friday.

Von: amqp@lists.oasis-open.org <amqp@lists.oasis-open.org> im Auftrag von Clemens Vasters <clemensv@microsoft.com>
Gesendet: Montag, Juni 8, 2020 5:05 PM
An: amqp@lists.oasis-open.org
Betreff: [amqp] AMQP Event Streams draft PR



This is the first discussion proposal for the event streams spec. I’m missing capability negotiation, the request/response call to retrieving partition information, and conformance clauses at this time.


I’ve tried to separate out a few concerns that are conflated in some of the event stream engines.


  1. Partitioning – I’m making partitioning optional, but provide a framework for when partitions exist. The goal is for a client to be able to use a partitioned store without necessarily knowing, but for the client to be explicit about picking partitions if it wants to.
  2. Delivery offsets/filters – picking offsets is also optional. If you attach a “receive” link without picking an offset, the behavior is as with subscribing to a regular queue broker subscription that you get messages made available for delivery after the link has been attached. Offsets reach back into the retained log. There are reserved offset identifiers for “earliest” and “latest”. The offset values themselves are opaque to clients, meaning they must always proceed with offset values they got from the engine. Optionally, you can reach into the log also with a time based offset. I am intentionally allowing for those source filters to be full AMQP filters, meaning that an implementation could quite well choose to combine reaching back into the retained log and regular filtering.
  3. Link ownership – the concept of “consumer groups” is really about coordinating ownership of partitions amongst a group of consumers. If the event log wants to coordinate ownership, it drops links for the parties that get re-associated and reassigns as they reconnect. If the consumers want to coordinate amongst themselves using some leader election scheme, they need for the event log to help breaking existing associations and “steal” links. That is what the described epoch model is for.


Functionally, Azure Event Hubs’ (and Azure IoT Hub’s) implementation is fairly close to what is described here, but I’ve made a few cleanup changes. For instance, Event Hubs puts the consumer group and partition information into the address path and I moved that into link properties. I also reinterpreted the SQL filter that Event Hubs uses as a restricted implementation of the AMQP Filter Expressions SQL filter. And while at it, I made the naming consistent and removed old-school prefixes like “x-opt”.


The difference to Apache Kafka’s partition ownership coordination model is that it is completely optional for the broker to be in the loop on consumer balancing because this interface is designed for an AMQP gateway that fronts a broker cluster. Therefore, clients can coordinate partition ownership completely amongst themselves in this model and assignments can be stable even if the broker rebalances at the back of the gateway. The gateway can also maintain connections and links through backend broker restarts/rebalances. At the same time, the gateway can also take ownership of partition ownership assignments and run a centralized balancing model. The protocol also supports a connection per partition directly to the brokers. I have yet to fit a mention of redirection where a client attaches indicating a request to own a particular partition and then being redirected to the owning broker.





Clemens Vasters

Messaging Platform Architect

Microsoft Azure

È+49 151 44063557

*  clemensv@microsoft.com   
European Microsoft Innovation Center GmbH | Gewürzmühlstrasse 11 | 80539 Munich| Germany
Geschäftsführer/General Managers: Keith Dolliver, Benjamin O. Orndorff 
Amtsgericht Aachen, HRB 12066



[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]