amqp message

Subject: Management notifications, question about filters

From: aconway <aconway@redhat.com>
To: amqp <amqp@lists.oasis-open.org>
Date: Fri, 23 Oct 2015 14:58:16 -0400

I have finally started working on implementing management notifications
so I will be updating the proposal with some real-life experience
shortly. I've re-attached the initial writeup in case you've forgotten
it.

My main concern right now is that several people in the know have
cautioned me against the use of JMS style selectors for notification
filtering. JMS selectors are are (and I quote)

* Syntactically overblown - although that's doable.
* Semantically complex - They use 3 value logic (inherited from SQL).
* No good specification - actually no real specification at all. Just a
few paragraphs in the JMS documentation that point you loosely at the 
1992(?) SQL spec.

However nobody has proposed a better alternative. I am very reluctant
to start inventing a new boolean expression language from scratch. So
does anybody have good/bad experience with selectors and think they're
a horrible/reasonable fit? Does anybody have alternative proposals for
a filtering language?

In the absence of better proposals I'm inclined to stick to selectors.
In spite of their warts they are known and implemented in JMS and by at
least some AMQP vendors. They provide a basic boolean expression
language which is all we need. If there is a better alternative though,
I'd like to hear about it.

AMQP Management Event Notification 0.2
======================================

Management clients often need to know when the state of a managed entity
has changed. This can be for monitoring or logging purposes, or to react
when something becomes ready for use.

A client can poll for changes with READ or QUERY operations. This works
well for clients that only need occasional updates, or want periodic
"snapshots" of the state without following every change.

However other clients need rapid notification to react to specific
events quickly. Here polling is inefficient. Polling too often overloads
the agent, polling too seldom causes delays. Some clients need to track
every event of a certain type, snapshots are not enough. In that case
polling doesn't work at all.

Event notification allows such clients to subscribe for continuous
notification of selected events as soon as the agent becomes aware of
them.

Event notification is an optional feature (TO-DO should it be
required?). If it is not supported, the GET-EVENTS operation MUST
receive a 501 (Not Implemented) response.

Event Types
-----------

An Event Type is a named set of attributes. The name is a case-sensitive
string. Implementers MAY define their own Event Types which MUST be
named using a reverse domain name prefix owned by the implementer, e.g.
"com.example.broker.queueDeleted".

Every Event Type MUST have an attribute "eventType" which carries the
Event Type name as a string. Implementers may define additional required
or optional attributes for their Event Types.

Event Types do not have operations, annotations or extensions. No
standard event types are defined by this specification (TO-DO should
they be?)

GET-EVENTS
~~~~~~~~~~

An additional operation GET-EVENTS to be added to org.amqp.management
node as follows.

Request
^^^^^^^

Additional application properties:

Body MUST be ignored.

Response
^^^^^^^^

If the request was successful then the statusCode MUST be 200 (OK) and
the body of the message MUST contain a ``map``. The keys in the map MUST
be the set of Management Event Types for which attribute names are being
provided. For any given key, the value MUST be a list of strings
representing the attribute names that this Management Event Type
possesses.

If event notification is not supported, the statusCode MUST be 501 (Not
Implemented).

The Notification Model
----------------------

This specification defines an abstract model of notification as follows.
Conceptually every event in the system corresponds to a "virtual
notification message" with full details of that event. The virtual
notification message subject MUST be the Event Type name. The
application-properties MUST include all required attributes and their
values for the Event Type, in particular they MUST include a property
"eventType" with the Event Type name. The application-properties MAY
include any optional attributes that are set for that event.

Client subscriptions can include a *filter* which selects a subset of
the virtual notification messages that the client is interested in. The
filter has access to the full event details in the virtual notification
message.

In addition a client subscription can specify a *transformation*. That
transformation can delete unwanted information from a message or cause
it to be re-written in a more compact form. The resulting *real*
notification messages are actually sent to the client.

An implementation MUST deliver real messages that are equivalent to what
*would have been* constructed by applying client filters and and
transformations to the full set of virtual messages. The implementation
is *not* required to actually construct all the virtual messages. It can
use any optimization techniques internally to avoid generating
unnecessary data or re-use shared data provided the real messages sent
to clients are consistent with the abstract model.

As described below implementations are also given wide latitude as to
what filtering or transformation capabilities they provide so simple or
sophisticated implementations are possible.

Subscribing and Filtering
-------------------------

To subscribe for notification messages, a client creates a link from the
address "$management".

The link may have a *filter* attached. The link behaves as if the filter was
applied to the *virtual notification message* containing all attributes and
values for the event as application-properties.

The filtering capabilities depend on the filters provided by the agent.
An agent SHOULD provide the APACHE.ORG:SELECTOR described in
http://www.amqp.org/specification/1.0/filters. This allows clients to
filter on arbitrary SQL-like expressions over all the attributes of the
event including "eventType".

Since the event type MUST also be included in the virtual message
subject, even simple filters (for example the legacy filters described
in http://www.amqp.org/specification/1.0/filters) MAY be used to filter
by event type.

Transforming Notifications (TO-DO needs clarification)
------------------------------------------------------

Event notifications may be high volume, so the size of the *real*
notification messages is important. Consider a client that wants to
monitor the size of a single queue. The virtual notification messages
for enqueue and dequeue events will contain the event type name (twice,
as subject and as a named application property), queue name, queue size,
possibly statistics like enqueue and dequeue rates - all with the
attribute name strings. This is a huge overhead for a client that only
needs a few integer values. Even if the client needs the eventType and
queueName for example, there is still a large overhead in carrying
redundant name strings "eventType", "queueName" and "queueSize" on every
message.

To resolve this problem a client may specify a *transformation* as well
as a filter.

TO-DO: how to attach the transformation? I think we can use the filter
slot and define a "transforming filter" which consists of a filter (as
already defined) plus a transformation. We can allow the set of possible
transformations to be open-ended, like the set of possible filters.
Other possible places to put the transformation include link properties
or terminus capabilities.

Implementations SHOULD implement the following simple "notification
transformation". It allows the subscriber to:

- control which attributes to include in the real message.
- request a compact representation that does not include attribute
names.

The transformation is specified as a map with the following entries:

*requestedAttributes* (required, list of strings): names of attributes
to be included. The real message is like the virtual message but
includes only the requested attributes in application-properties.

*compact* (optional, boolean): If present and true the real message is
re-written with *no* application-properties and *no* subject. The
message body is a list of attribute values in the order of
*requestedAttributes*. Missing or undefined attributes are represented
by NULL values. There are no attribute names in the message, the meaning
of the values is implied by their position in the list.

If a filter allows multiple event types, then can be ambiguity if an
attribute of the same name exists on different event types with
different meanings. In that case the client can include ``eventType`` in
*requestedAttributes* to disambiguate. However if there is no ambiguity
the client is not obliged to include ``eventType`` as it is needless
overhead.

Note a client is not required to specify any filter or transformation.
Without a transformation, the client will receive the virtual
notification message as a real message. Without a filter the client will
receive the entire virtual notification stream as a real message stream.

--------------

Notes and questions, not part of the spec:
==========================================

Denial of Service
-----------------

Defending against deliberate DOS attacks is a job for a separate
security spec. For notification it requires a security model that can do
things like restrict access to notification generally, restrict access
by event type, and impose quotas on the rate or volume of traffic
allowed to different security roles. This is not just a notification
issue: you can DOS attack management with query bombing or AMQP level
connection flooding. So a security model would have to cover all of
that.

However notification is different from GET or QUERY in that it is easier
to *accidentally* load the node or network with a badly written or
forgotten filter or transformation. I'm not sure if there's anything we
can do about that. Past systems I've worked on with similar notification
mechanisms (e.g. Qpid C++ broker) don't have anything to defend against
such accidents, in practice this seems to work out OK.

Custom Alerts
-------------

e.g. send me a notification if the queue size goes above X where each
subscriber had a different X

One possibility is to push this to the client:

1. client subscribes for "eventType=enqueue AND queue=Q AND size > X"
2. client sends GET(Q) to get current state of queue (in case the queue is
already > X but nobody's enquing)
3. client triggers if response to GET has size > X or if it gets an event on
its subscription.
4. client cancels the subscription.

This is not trivial but could easily be encapsulated in client-side
libraries so it is easy for the user. [This is similar to the
request-response patterns described in the main spec - they are not
trivial for clients but are easily encapsulated in libraries.] My
feeling is this is better handled on the client side, as otherwise we
are going to end up holding conversational state on the management node
for every client.

Justification for some design decisions
---------------------------------------

This design does not allow multiple events to be batched in a single
message. IMO an implementation should batch messages at the transport
(into TCP packets) NOT batch events into a message. Batching in messages
complicates the client, defeats transport level batching (by forcing
"fat" messages on the transport) and rules out (or badly complicates)
the use of AMQP filters to select events.

I considered an alternative design where the client sends a subscription
request with a reply-to address and the agent then sends notification
messages to the reply-to address.

I rejected that design for the following reasons:

1. It is impossible in general to tell when such a subscriber goes away
if it crashes or fails to send an explicit unsubscribe message.
Implementations must already deal with closing links when clients
disconnect or heartbeat out so using links solves this problem. Even
in an indirect topology (like link-routed Qpid dispatch networks) the
routers must proxy link closure to be AMQP-correct.

2. We should use AMQP standard features to solve AMQP problems, not
re-invent the wheel in a form only useful for management.
Filters/selectors were designed for exactly this purpose we should
make them work for us.

Standard event types
--------------------

There is scope for defining standard events from the AMQP performatives.
Performatives like link-open, connection-open, disposition etc. all
signify an event that might be of interest to a management console.
Probably best to finalize the general event model before trying to
standardize events.

----

Counting and skipping
---------------------

Should we allow a client to subscribe for a limited number of responses and then
automatically unsubscribe? The trigger pattern described above would be more
efficient and easier to implement if the client can request a single event and
auto-unsubscribe.

TODO: terms "every-n-events alert", "one-shot"