Re: [amqp] [management] Proposal for event notifications (initial outline)

[Alan Conway <aconway@redhat.com>]

On Mon, 2015-02-16 at 20:05 +0100, Jakub.Scholz@deutsche-boerse.com
wrote:
> Hi Alan,
> 
> If I understand it correctly, this basically introduces two distinct
> implementations of the events - I don't think that's a good idea. 

Agreed, I'll drop the "with names" format. It's very inefficient and not
necessary if we have GET-EVENTS.

>  Also,
> your compact representation allows every link to define its own list of
> attributes. So in theory you can have N links, each with different
> attributeNames defined and you will have to generate and publish the events
> N times in N different formats.
> 
> What about having just one implementation which stands somewhere in the
> middle:
> 1) GET-EVENTS returns the list of attributes for every event type
> 2) The broadcasted events will have only a list of values in the same order
> as was the order of attributes in the GET-EVENTS response - so we save the
> attribute names.
> 
> This will not be as effective on the wire, but it would allow to have for
> every event only one message regardless the number of subscribers.

Not sure about this. In unicast networks I expect the in-memory
construction of the messages will be less important than the network
bandwidth cost of sending them (depends on hardware of course.) Allowing
clients to select a subset of attributes may reduce bandwidth more than
sending the entire attribute set on each link each, even if it saves
some in-memory work to construct multiple messages.

In a broadcast or multi-cast network of course the trade-off is
different but I think the AMQP protocol right now really assumes unicast
network connections, and would need some changes or extensions to handle
multicast.

> Thanks & Regards
> Jakub
> 
> <amqp@lists.oasis-open.org> wrote on 16/02/2015 17:24:10:
> 
> > From: Alan Conway <aconway@redhat.com>
> > To: "John O'Hara" <john@rjohara.com>,
> > Cc: amqp@lists.oasis-open.org, "Arrott, Matt"
> > <marrott@ucsd.edu>, "Godfrey, Robert X"
> > <robert.godfrey@jpmorgan.com>
> > Date: 16/02/2015 17:24
> > Subject: Re: [amqp] [management] Proposal for event
> > notifications (initial outline)
> > Sent by: <amqp@lists.oasis-open.org>
> >
> >
> > I have a couple of thoughts for extending the event notification
> > proposal, the original is below for reference. I'll put them all
> > together if after any feedback.
> >
> > *Event meta-data*: Add a GET-EVENTS operation to the
> > "self" entity (like
> > existing GET-ATTRIBUTES) to get a map of event-type name
> > to attribute
> > name list.
> >
> >
> > *Compact event representation*: The current proposal is
> > very fat on the
> > wire - attribute names are repeated in every notification
> > and there's no
> > way to select a subset of attributes. I suggest the
> > following addition
> > inspired by the existing QUERY response:
> >
> > When subscribing for events the link attach properties MAY
> > contain the
> > key `attributeNames` with a list of attribute names. If so, event
> > messages are tested against the filter as before, but are sent in a
> > compact form. Attribute names/values are not sent in the application
> > properties, instead the body contains a list of attribute
> > values in the
> > same order as `attributeNames`. Missing attributes are
> > represented by
> > NULL values.
> >
> > Without `attributeNames` you get the names and values. This is less
> > efficient but may be easier to work with for dynamic or exploratory
> > tools that can interact with many different implementations. With
> > `attributeNames` you get just what you ask for with no
> > overhead. This is
> > better for tools that know what they want, and in heavy
> > traffic systems
> > where the overhead of attribute names is excessive.
> >
> >
> > *Standard event types*: We could probably generate a
> > reasonable set of
> > standard event types directly from the performatives in
> > the AMQP spec.
> > Every performative (link-open, connection-open,
> > disposition etc.) does
> > signify and event that might be of interest to a
> > management console. Is
> > it too early to think about standardizing this?  Are there
> > any pitfalls?
> >
> >
> >
> > Original proposal
> > -----------------
> >
> > # AMQP Management Event Notification
> >
> > Management clients often need to know when the state of a
> > managed entity
> > has changed.  This can be for monitoring or logging purposes, or to
> > react when something becomes ready for use.
> >
> > Polling the management agent for changes with READ or
> > QUERY operations
> > is inefficient. Clients either poll too often, which overloads the
> > agent, or too seldom which causes needless delays in the system.
> >
> > Event notification allows management clients to subscribe
> > for continuous
> > notification of selected events as soon as the agent
> > becomes aware of
> > them.
> >
> > ## Event Types
> >
> > An Event Type is a named set of attributes. The name is a
> > case-sensitive
> > string. Implementers MAY define their own Event Types which MUST be
> > named using a reverse domain name prefix owned by the
> > implementer, e.g.
> > "com.example.broker.queueDeleted".
> >
> > Every Event Type MUST have an attribute "eventType" which
> > carries the
> > Event Type name as a string. Implementers may define
> > additional required
> > or optional attributes for their Event Types.
> >
> > Event Types do not have operations, annotations or extensions. No
> > standard event types are defined by this specification.
> >
> > ## Notification Messages
> >
> > A notification message contains a single event. The
> > message subject MUST
> > be the Event Type name.  The application-properties MUST include all
> > required attributes for that Event Type, in particular
> > "eventType" with
> > the Event Type name.  They MAY include any optional
> > attributes for the
> > Event Type.
> >
> > ## Subscribing for Event Notification
> >
> > To subscribe for notification messages, a client creates a
> > link from the
> > address "$management.events" with a *filter* (TODO xref)
> > describing the
> > events that it is interested in.  Closing the link ends the
> > subscription.
> >
> > The filtering capabilities depend on the filters provided
> > by the agent.
> > An agent SHOULD provide the APACHE.ORG:SELECTOR described in
> > <http://www.amqp.org/specification/1.0/filters>.  This
> > allows clients to
> > filter on arbitrary SQL-like expressions over all the
> > attributes of the
> > event including "eventType".
> >
> > Since the event type MUST also be included in the message
> > subject, even
> > simple filters (for example the legacy filters described in
> > <http://www.amqp.org/specification/1.0/filters>) MAY be
> > used to filter
> > by event type.
> >
> > Any filter that can operate on the subject or application-
> > properties of
> > a message will be able to work with event notification
> > messages, but for
> > flexibility and interoperability implementers SHOULD provide
> > APACHE.ORG:SELECTOR.
> >
> > ----
> >
> > # Notes, not part of the spec:
> >
> > Needs cross references to relevant sections on filters etc.
> >
> > This design does not allow multiple events to be batched in a single
> > message. IMO an implementation should batch messages at
> > the transport
> > (into TCP packets) NOT batch events into a message.
> > Batching in messages
> > complicates the client, defeats transport level batching (by forcing
> > "fat" messages on the transport) and rules out (or badly
> > complicates)
> > the use of AMQP filters to select events.
> >
> > I considered an alternative design where the client sends
> > a subscription
> > request with a reply-to address and the agent then sends
> > notification
> > messages to the reply-to address.
> >
> > I rejected that design for the following reasons:
> >
> > 1. It is impossible in general to tell when such a
> > subscriber goes away
> > if it crashes or fails to send an explicit unsubscribe message.
> > Implementations must already deal with closing links when clients
> > disconnect or heartbeat out so using links solves this
> > problem. Even in
> > an indirect topology (like link-routed Qpid dispatch networks) the
> > routers must proxy link closure to be AMQP-correct.
> >
> > 2. We should use AMQP standard features to solve AMQP problems, not
> > re-invent the wheel in a form only useful for management.
> > Filters/selectors were designed for exactly this purpose
> > we should make
> > them work for us.
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe from this mail list, you must leave the
> > OASIS TC that
> > generates this mail.  Follow this link to all your TCs in OASIS at:
> > https://www.oasis-open.org/apps/org/workgroup/portal/
> > my_workgroups.php
> >
> 
> 
> ----------------------------------------------------------------------------
> 
> Deutsche Börse Services s.r.o.
> Managing Directors/Geschäftsführung:
> Michael Gassmann, Mats Andersson.
> Limited liability company with registered office at
> Sokolovská 662/136B, CZ-186 00 Prague 8
> recorded in the Commercial Register IC: 275 77 015.
> Maintained by the city court in Prague,
> Sec. C, File No. 116874.
> 
> -----------------------------------------
> Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen.
> Wenn Sie nicht der beabsichtigte Empfaenger sind, informieren Sie bitte
> sofort den Absender und loeschen Sie diese E-Mail. Das unbefugte Kopieren
> dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen
> ist nicht gestattet.
> 
> The information contained in this message is confidential or protected by
> law. If you are not the intended recipient, please contact the sender and
> delete this message. Any unauthorised copying of this message or
> unauthorised distribution of the information contained herein is prohibited.
> 
> Legally required information for business correspondence/
> Gesetzliche Pflichtangaben fuer Geschaeftskorrespondenz:
> http://deutsche-boerse.com/letterhead
>