I guess we do agree on the queuing pattern. :-) Calling the queuing conduit a consumer gave me an
impression that the message distribution from the queue was left out
to applications behind the consumer and ordinary pub/sub would be
sufficient to cover all messaging.
I also agree that the messaging
distribution in this case is orthogonal to distribution defined by
topics.
Lily
Lily Liu wrote:
I
like your taxonomy that sorts the issues
out nicely.
The
only items I would add
to "Determined by Subscriber" category are subscription hints (such as
getting messages by publisher order, or by priority), and durable
subscribers.
I lumped
durable subscribers under QoS/Durability in the face of process failures, but
it might be worth calling out specifically. The subscription hints are
very worthwhile additions (and exactly the sort of thing I'd want to handle
via an extension point).
I still have a different view on "1-of-N delivery" (Issue WSN 2.26) . To me, message queue is
not a special case of pub/sub with only one consumer involved. Message
queue is a different delivery pattern that indicates messages are
sent through a single
pipe. These messages can be picked up by a single consumer or
multiple consumers. Multiple consumers of a message queue may get the
messages from the queue by a first come
first served manner, or by a more complicated algorithm. Message
queue enables shared state consumers, load balance, message priority, and other queuing
functions. These are common messaging
features that should be surfaced through the WSN
interface. Funny, that sounds
almost exactly like my view of the situation :-).
I think the key here
is that there are two equally valid views of "consumer". As indicated by
the phrase "single pipe", at some point, messages are getting put into
a single conduit. On the other hand, they are getting pulled out of the
conduit by multiple parties. Either the single pipe or the multiple
parties (or both) could be termed "consumers".
The relevant technical
question is how to specify this when making a subscription. I would like
to do this in a composable way, so that notifications could be sent, say
- To consumer A, and also to
- One of
- Consumer B
- Consumer C, and also one of
In other words
- A gets every notification
- B and C together get every notification (and no message goes to
both)
- D and E together get every notification C gets (and no message goes to
both)
This sort of thing is easy to put together with a 1-of-N
fanout module, and viewed this way it's easy to change the picture
dynamically. You could also describe it in terms of dependencies (the in
other words part), but I'm not sure that that will handle dynamic changes as
gracefully.
Also, it seems that such matters of plumbing have nothing
at all to do with topics. The ability to split a sequence of messages
among several recipients seems useful in itself.
One
theme I've been harping on lately is the need to separate the concerns of
what data to send from the mechanics of how that data gets
delivered. While I still believe that some form of this distinction
will be essential, I would like to step back a bit and try to get a broad
look at all the various parameters to be decided when establishing
a subscription, whether they are currently written into WSN or WSE, or are
currently considered negotiable by policy, or are not yet clearly
classified at all.
It occurs to me that one criterion for deciding
how to treat these parameters is which party or parties have a say in
them. For example, the NotificationProducer determines what
notifications it can possibly support, while the Subscriber decides what
part of that universe it's actually interested in.
One reason that
delivery mechanisms appear to behave differently is that more than one
party may have a say in them. A Consumer may prefer certain ways of
getting data, but the NotificationProducer may not support the Consumer's
favorite means. The Consumer cannot dictate the means of delivery
because it doesn't know what the NotificationProducer can support.
The NotificationProducer could dictate the means of delivery, but
shouldn't, since it doesn't know the Consumer's preferences.
Instead, we need some means of conflict resolution, whether negotiation, a
resolution algorithm, or some combination of the two. In simpler
cases, for example what set of topics does a producer support, it's enough
for the owning party to be able to advertise and for other parties to be
able to make queries.
So here is a preliminary list of variable
parameters for a given notification deployment, with notes as to who the
stakeholders are. I've tried to incorporate the policy issues we've
unearthed from the issues list, along with anything else of interest, but
I've almost certainly missed important items. I'm hoping that
presenting a taxonomy like this will help flush out missing items, and
also provide a place to hang the ones that turn up for whatever
reason. I apologize if this is a little cryptic. Many of these
issues have long discussions and quite a bit of context behind
them.
- The universe of possible notifications. Determined by the
NotificationProducer.
- What data to send. Determined by Subscriber
- Topic filtering
- Precondition filtering
- Selector filtering
- Submessage selection (Issue 2.30 from WSDM)
- How to deliver the data. Negotiated by producer and consumer.
- Pull vs. push delivery (Issue 1.4)
- QoS
- Delivery guarantees
- Queuing and replaying
- Durability in the face of process failures
- Continuity in the face of subscription modifications (Issue
2.28)
- Security (See note 1 below)
- DDOS mitigation (e.g., double opt-in) (Issue 2.6)
- 1-of-N delivery (Issue 2.26. See note 2 below)
- queuing and replay (Issue 2.27)
- boxcarring of messages
- envelope data (subscription ID, etc. See note 3 below).
- <>message format (Issue 2.13, 3.2. See note 4
below).
- Administration. NotificationProducer has a say. Who
else?
- Means of garbage collection (Issue 2.18: scheduled termination vs.
other means).
- Means of notification of subscription life events (See note 5
below)
- Security of administration (See note 1 below)
- Uncategorized
- GetCurrentMessage vs. Initial value/update vs. stateless (See note
6 below)
- Authorization to make subscriptions on behalf of a
Consumer.
Notes:
- In the delivery context, security refers to the usual concerns of
encryption, signatures and so forth. There are also separate
issues such as who may make a subscription on whose behalf and who may
delete, pause, resume or modify a subscription, or who may receive
notice of subscription life events.
- I would provisionally regard 1-of-N delivery as delivery to a single
consumer representing the message queue, with the actual recipients
attached to the queue without the NotificationProducer knowing anything
about them, but I realize this issue is still open.
- It is currently configurable whether a Notify message contains a
subscription reference, and even whether Notify is used for delivery at
all.
- This is drawn from the issues list. I'm not entirely sure what
level the issue is at, but it may have to do with choosing between
transport-level options -- as a hypothetical example, whether messages
are to be compressed by some means. Boxcarring is at least a
related issue.
- WSE allows for an explicit callback address for subscription
death. WSN treats this as a special case of notification of a
resource change. I've chosen "subscription life events" as one
might be interested not only in subscription death but also
modifications. Some systems also support notification of
subscription creation, that is, notifications of the form "tell me when
anyone creates a subscription on this topic".
- A while ago I wrote about GetCurrentMessage being a special case of
"get the current state of the resource," with the understanding that
notifications may reflect only state changes and not the entire
state. For example, it may be possible to query a large dataset
and then be notified of presumably much smaller changes to that
dataset. It would be good to be able to advertise this sort of
semantic.
|