AMQP event notification v0.2

[Alan Conway <aconway@redhat.com>]

Updated proposal on event notification. This probably does not address
all the issues raised on the list since the last proposal but it is
better filled out so hopefully can re-focus the discussion - feel free
to re-raise your concerns if they are not addressed here.

Cheers,
Alan.

AMQP Management Event Notification 0.2
======================================

Management clients often need to know when the state of a managed entity has changed.  This can be for monitoring or logging purposes, or to react when something becomes ready for use.

A client can poll for changes with READ or QUERY operations. This works well for clients that only need occasional updates, or want periodic "snapshots" of the state without following every change.

However other clients need rapid notification to react to specific events quickly. Here polling is inefficient. Polling too often overloads the agent, polling too seldom causes delays. Some clients need to track every event of a certain type, snapshots are not enough. In that case polling doesn't work at all.

Event notification allows such clients to subscribe for continuous notification of selected events as soon as the agent becomes aware of them.

Event notification is an optional feature (TO-DO should it be required?). If it is not supported, the GET-EVENTS operation MUST receive a 501 (Not Implemented) response.

## Event Types

An Event Type is a named set of attributes. The name is a case-sensitive string. Implementers MAY define their own Event Types which MUST be named using a reverse domain name prefix owned by the implementer, e.g. "com.example.broker.queueDeleted".

Every Event Type MUST have an attribute "eventType" which carries the Event Type name as a string. Implementers may define additional required or optional attributes for their Event Types.

Event Types do not have operations, annotations or extensions. No standard event types are defined by this specification (TO-DO should they be?)

### GET-EVENTS

An additional operation GET-EVENTS to be added to org.amqp.management node as follows.

#### Request

Additional application properties:

| Key         | Type   | Mandatory | Description                                                 |
| `eventType` | string | No        | If set, return attributes only for the specified event type |

Body MUST be ignored.

#### Response

If the request was successful then the statusCode MUST be 200 (OK) and the body of the message MUST contain a `map`. The keys in the map MUST be the set of Management Event Types for which attribute names are being provided. For any given key, the value MUST be a list of strings representing the attribute names that this Management Event Type possesses.

If event notification is not supported, the statusCode MUST be 501 (Not Implemented).

## The Notification Model

This specification defines an abstract model of notification as follows. Conceptually every event in the system corresponds to a "virtual notification message" with full details of that event. The virtual notification message subject MUST be the Event Type name.  The application-properties MUST include all required attributes and their values for the Event Type, in particular they MUST include a property "eventType" with the Event Type name.  The application-properties MAY include any optional attributes that are set for that event.

Client subscriptions can include a *filter* which selects a subset of the virtual notification messages that the client is interested in. The filter has access to the full event details in the virtual notification message.

In addition a client subscription can specify a *transformation*. That transformation can delete unwanted information from a message or cause it to be re-written in a more compact form. The resulting *real* notification messages are actually sent to the client.

An implementation MUST deliver real messages that are equivalent to what *would have been* constructed by applying client filters and and transformations to the full set of virtual messages. The implementation is *not* required to actually construct all the virtual messages. It can use any optimization techniques internally to avoid generating unnecessary data or re-use shared data provided the real messages sent to clients are consistent with the abstract model.

As described below implementations are also given wide latitude as to what filtering or transformation capabilities they provide so simple or sophisticated implementations are possible.

## Subscribing and Filtering

To subscribe for notification messages, a client creates a link from the address "$events" (TO-DO: this address is a placeholder, need to consider addressing spec, wild-card addressing.) with a *filter* attached to the link. The link behaves as if the filter was applied to the *virtual notification message* containing all attributes and values for the event as application-properties.

The filtering capabilities depend on the filters provided by the agent.  An agent SHOULD provide the APACHE.ORG:SELECTOR described in <http://www.amqp.org/specification/1.0/filters>.  This allows clients to filter on arbitrary SQL-like expressions over all the attributes of the event including "eventType".

Since the event type MUST also be included in the virtual message subject, even simple filters (for example the legacy filters described in <http://www.amqp.org/specification/1.0/filters>) MAY be used to filter by event type.

## Transforming Notifications (TO-DO needs clarification)

Event notifications may be high volume, so the size of the *real* notification messages is important. Consider a client that wants to monitor the size of a single queue. The virtual notification messages for enqueue and dequeue events will contain the event type name (twice, as subject and as a named application property), queue name, queue size, possibly statistics like enqueue and dequeue rates - all with the attribute name strings. This is a huge overhead for a client that only needs a few integer values. Even if the client needs the eventType and queueName for example, there is still a large overhead in carrying redundant name strings "eventType", "queueName" and "queueSize" on every message.

To resolve this problem a client may specify a *transformation* as well as a filter.

TO-DO: how to attach the transformation? I think we can use the filter slot and define a "transforming filter" which consists of a filter (as already defined) plus a transformation. We can allow the set of possible transformations to be open-ended, like the set of possible filters. Other possible places to put the transformation include link properties or terminus capabilities.

Implementations SHOULD implement the following simple "notification transformation". It allows the subscriber to:

- control which attributes to include in the real message.
- request a compact representation that does not include attribute names.

The transformation is specified as a map with the following entries:

*requestedAttributes* (required, list of strings): names of attributes to be included. The real message is like the virtual message  but includes only the requested attributes in application-properties.

*compact* (optional, boolean): If present and true the real message is re-written with *no* application-properties and *no* subject. The message body is a list of attribute values in the order of *requestedAttributes*. Missing or undefined attributes are represented by NULL values. There are no attribute names in the message, the meaning of the values is implied by their position in the list.

If a filter allows multiple event types, then can be ambiguity if an attribute of the same name exists on different event types with different meanings. In that case the client can include `eventType` in *requestedAttributes* to disambiguate. However if there is no ambiguity the client is not obliged to include `eventType` as it is needless overhead.

Note a client is not required to specify any filter or transformation. Without a transformation, the client will receive the virtual notification message as a real message. Without a filter the client will receive the entire virtual notification stream as a real message stream. 

----

# Notes and questions, not part of the spec:

## Denial of Service

Defending against deliberate DOS attacks is a job for a separate security spec. For notification it requires a security model that can do things like restrict access to notification generally, restrict access by event type, and impose quotas on the rate or volume of traffic allowed to different security roles. This is not just a notification issue: you can DOS attack management with query bombing or AMQP level connection flooding. So a security model would have to cover all of that.

However notification is different from GET or QUERY in that it is  easier to *accidentally* load the node or network with a badly written or forgotten filter or transformation. I'm not sure if there's anything we can do about that. Past systems I've worked on with similar notification mechanisms (e.g. Qpid C++ broker) don't have anything to defend against such accidents, in practice this seems to work out OK.

## Custom Alerts

e.g. send me a notification if the queue size goes above X where each subscriber had a different X

One possibility is to push this to the client:
1. client subscribes for "eventType=enqueue AND queue=Q AND size > X"
2. client sends GET(Q) to get current state of queue (in case the queue is already > X but nobody's enquing)
3. client triggers if response to GET has size > X or if it gets an event on its subscription.
4. client cancels the subscription.

This is not trivial but could easily be encapsulated in client-side libraries so it is easy for the user. [This is similar to the request-response patterns described in the main spec - they are not trivial for clients but are easily encapsulated in libraries.] My feeling is this is better handled on the client side, as otherwise we are going to end up holding conversational state on the management node for every client.

## Justification for some design decisions

This design does not allow multiple events to be batched in a single message. IMO an implementation should batch messages at the transport (into TCP packets) NOT batch events into a message. Batching in messages complicates the client, defeats transport level batching (by forcing "fat" messages on the transport) and rules out (or badly complicates) the use of AMQP filters to select events.

I considered an alternative design where the client sends a subscription request with a reply-to address and the agent then sends notification messages to the reply-to address.

I rejected that design for the following reasons:

1. It is impossible in general to tell when such a subscriber goes away if it crashes or fails to send an explicit unsubscribe message.  Implementations must already deal with closing links when clients disconnect or heartbeat out so using links solves this problem. Even in an indirect topology (like link-routed Qpid dispatch networks) the routers must proxy link closure to be AMQP-correct.

2. We should use AMQP standard features to solve AMQP problems, not re-invent the wheel in a form only useful for management.  Filters/selectors were designed for exactly this purpose we should make them work for us.


## Standard event types

There is scope for defining standard events from the AMQP performatives. Performatives like link-open, connection-open, disposition etc. all signify an event that might be of interest to a management console. Probably best to finalize the general event model before trying to standardize events.