Lily Liu wrote:
I guess we do agree on the queuing
pattern. :-) Calling the queuing conduit
a consumer gave me an impression that the message distribution from the
queue was left out to applications behind the consumer and ordinary
pub/sub would be sufficient to cover all messaging.
I think we're converging. I didn't mean that an ordinary consumer
would be responsible for distributing messages from a queue. I had a
scenario more like brokering in mind, where the distributor plays the
role of a consumer from the point of view of the NotificationProducer.
We need to be very careful here to separate semantics from
implementation. I believe the semantics come out simplest if we define
them as though there is a separate distributor acting as an
intermediary, consuming from the NotificationProduer and producing for
the ultimate Consumers.
That doesn't mean that such a process has to be deployed. The
NotificationProducer could just as well take everything on itself. The
key question is not how processes get deployed, but what messages have
to be passed.
Thinking aloud now, it would be sufficient if the EPR returned by the
subscribe request also acted (mostly) like a Notification Producer.
The message flow would be
- Subscribe to a topic, get back an EPR for the distributor.
- To join the queue, send a subscribe request to the distributor,
get back an EPR for that subscription.
- To leave the queue, destroy the subscription
- The distributor EPR is also an ordinary subscription. For
example, to shut down the queue entirely, just cancel the main
subscription.
Joining/leaving looks much like regular subscription, except that there
is no topic involved.
I also
agree that the messaging distribution in this case is orthogonal to
distribution defined by topics.
To me it's a major open question how best to split these apart.
Lily Liu wrote:
I
like your taxonomy that sorts the issues
out nicely.
The
only items I would add to "Determined by Subscriber" category are subscription hints
(such as getting messages by publisher order, or by priority), and
durable subscribers.
I lumped durable subscribers under QoS/Durability in the face of
process failures, but it might be worth calling out specifically. The
subscription hints are very worthwhile additions (and exactly the sort
of thing I'd want to handle via an extension point).
I still have a different
view on "1-of-N delivery" (Issue WSN 2.26) . To me, message queue is not a
special case of pub/sub with only one consumer involved. Message
queue is a different delivery pattern that indicates messages are sent through a single pipe. These
messages can be picked up by a single consumer or multiple consumers.
Multiple consumers of a message queue may get the messages from the
queue by a first come first
served manner, or by a more complicated algorithm. Message queue
enables shared state consumers, load balance, message priority, and other queuing
functions. These are common messaging features
that should be surfaced through the WSN interface.
Funny, that sounds almost exactly like my view of the situation :-).
I think the key here is that there are two equally valid views of
"consumer". As indicated by the phrase "single pipe", at some
point, messages are getting put into a single conduit. On the other
hand, they are getting pulled out of the conduit by multiple parties.
Either the single pipe or the multiple parties (or both) could be
termed "consumers".
The relevant technical question is how to specify this when making a
subscription. I would like to do this in a composable way, so that
notifications could be sent, say
- To consumer A, and also to
- One of
- Consumer B
- Consumer C, and also one of
In other words
- A gets every notification
- B and C together get every notification (and no message goes
to both)
- D and E together get every notification C gets (and no
message goes to both)
This sort of thing is easy to put together with a 1-of-N fanout module,
and viewed this way it's easy to change the picture dynamically. You
could also describe it in terms of dependencies (the in other words
part), but I'm not sure that that will handle dynamic changes as
gracefully.
Also, it seems that such matters of plumbing have nothing at all to do
with topics. The ability to split a sequence of messages among several
recipients seems useful in itself.
One theme I've been harping on lately is the need to separate the
concerns of what data to send from the mechanics of how that data gets
delivered. While I still believe that some form of this distinction
will be essential, I would like to step back a bit and try to get a
broad look at all the various parameters to be decided when
establishing a subscription, whether they are currently written into
WSN or WSE, or are currently considered negotiable by policy, or are
not yet clearly classified at all.
It occurs to me that one criterion for deciding how to treat these
parameters is which party or parties have a say in them. For example,
the NotificationProducer determines what notifications it can possibly
support, while the Subscriber decides what part of that universe it's
actually interested in.
One reason that delivery mechanisms appear to behave differently is
that more than one party may have a say in them. A Consumer may prefer
certain ways of getting data, but the NotificationProducer may not
support the Consumer's favorite means. The Consumer cannot dictate the
means of delivery because it doesn't know what the NotificationProducer
can support. The NotificationProducer could dictate the means of
delivery, but shouldn't, since it doesn't know the Consumer's
preferences. Instead, we need some means of conflict resolution,
whether negotiation, a resolution algorithm, or some combination of the
two. In simpler cases, for example what set of topics does a producer
support, it's enough for the owning party to be able to advertise and
for other parties to be able to make queries.
So here is a preliminary list of variable parameters for a given
notification deployment, with notes as to who the stakeholders are.
I've tried to incorporate the policy issues we've unearthed from the
issues list, along with anything else of interest, but I've almost
certainly missed important items. I'm hoping that presenting a
taxonomy like this will help flush out missing items, and also provide
a place to hang the ones that turn up for whatever reason. I apologize
if this is a little cryptic. Many of these issues have long
discussions and quite a bit of context behind them.
- The universe of possible notifications. Determined by
the NotificationProducer.
- What data to send. Determined by Subscriber
- Topic filtering
- Precondition filtering
- Selector filtering
- Submessage selection (Issue 2.30 from WSDM)
- How to deliver the data. Negotiated by producer and
consumer.
- Pull vs. push delivery (Issue 1.4)
- QoS
- Delivery guarantees
- Queuing and replaying
- Durability in the face of process failures
- Continuity in the face of subscription
modifications (Issue 2.28)
- Security (See note 1 below)
- DDOS mitigation (e.g., double opt-in) (Issue 2.6)
- 1-of-N delivery (Issue 2.26. See note 2 below)
- queuing and replay (Issue 2.27)
- boxcarring of messages
- envelope data (subscription ID, etc. See note 3
below).
- <>message format (Issue 2.13, 3.2. See note 4
below).
- Administration. NotificationProducer has a say. Who
else?
- Means of garbage collection (Issue 2.18: scheduled
termination vs. other means).
- Means of notification of subscription life events
(See note 5 below)
- Security of administration (See note 1 below)
- Uncategorized
- GetCurrentMessage vs. Initial value/update vs.
stateless (See note 6 below)
- Authorization to make subscriptions on behalf of a
Consumer.
Notes:
- In the delivery context, security refers to the usual
concerns of encryption, signatures and so forth. There are also
separate issues such as who may make a subscription on whose behalf and
who may delete, pause, resume or modify a subscription, or who may
receive notice of subscription life events.
- I would provisionally regard 1-of-N delivery as delivery
to a single consumer representing the message queue, with the actual
recipients attached to the queue without the NotificationProducer
knowing anything about them, but I realize this issue is still open.
- It is currently configurable whether a Notify message
contains a subscription reference, and even whether Notify is used for
delivery at all.
- This is drawn from the issues list. I'm not entirely
sure what level the issue is at, but it may have to do with choosing
between transport-level options -- as a hypothetical example, whether
messages are to be compressed by some means. Boxcarring is at least a
related issue.
- WSE allows for an explicit callback address for
subscription death. WSN treats this as a special case of notification
of a resource change. I've chosen "subscription life events" as one
might be interested not only in subscription death but also
modifications. Some systems also support notification of subscription
creation, that is, notifications of the form "tell me when anyone
creates a subscription on this topic".
- A while ago I wrote about GetCurrentMessage being a
special case of "get the current state of the resource," with the
understanding that notifications may reflect only state changes and not
the entire state. For example, it may be possible to query a large
dataset and then be notified of presumably much smaller changes to that
dataset. It would be good to be able to advertise this sort of
semantic.
|