Re: [wsn] Pull-type notification

[Matthew I Roberts <matt.roberts@uk.ibm.com>]

William,

I agree with many of your comments regarding the "pull-type" proposal -
deferring the buffer size/utilization to a policy statement seems to be a
good way of keeping the interfaces simple while allowing vendors the
ability to implement their preferred storage techniques. As with all these
behaviours we need to keep one eye on ensuring that we don't prohibit any
particular approach, and that we have suitable points to advertise these
policies, but I believe we can easily cover that in this case.

Moving on to your three suggested changes I had a different assumption on
what the semantics for GetMessage should be;

Quoting William;
> The first change might actually just be a clarification since I am not
> sure how Steve intended it to work. If there are 100 messages in the
> queue and I do a GetMessages(40) then I will receive the 40 most recent
> messages and the entire queue will be cleared. Basically, every
> GetMessages request clears the entire queue in addition to getting some
> or all of the most recent messages.

I interpretted Steve's proposal to be that any messages that were not
delivered as a result of a GetMessages(n) call would remain available until
the next time rather than being wiped clean as you suggest. This would
exactly parallel the JMS synchronous receive operations and enables the
application to handle notifications a few at a time rather than have an
imposed 'use it or lose it' style operation. A risk with your approach is
that you can never guarantee that the application can get every message (a
requirement for which I believe there are valid use cases).

A second issue is whether GetMessages(40) should return the 40 oldest
messages (traditional FIFO queue) or the 40 newest messages (FILO, which is
your proposal). I would prefer a traditional FIFO queue so that we get the
40 oldest messages (with no clean afterwards) as this provides the lossless
durable delivery of existing messaging systems like JMS. If a vendor wished
to implement the get-with-clean or FILO behaviours you suggest I think it
should be on the basis of a policy statement. Note that get-with-clean and
FILO (get newest) should probably always be combined, since not doing so
results in shuffling the messages into a different order.


Your second change was as below;

Quoting William;
> The second change is to have the GetMessages request contain 2 optional
> elements (or attributes): MaxMessages, as currently defined, but also
> OldestTime. If MaxMessages is present then don't send me any more than
> this many messages. If OldestTime is present then don't send me any
> message older than this date. The two filters can exist on their own or
> co-exist. Yes, they are transitive.

MaxMessages sounds like a useful and sensible attribute to provide on the
GetMessages call (subject to my suggestions above). However I am not aware
that we define the 'age' of an event as part of WSN, and the OldestTime
attribute seems to be have an implicit requirement here to timestamp all
messages. Clearly many existing messaging systems do have the concept of a
timestamp for all messages, however I would prefer to remove OldestTime
from the basic interface definition in favour of an optional policy.


Your third suggestion was as follows;

Quoting William;
> The last change is to create a special topic called "MessagesDropped".
> The payload for this topic contains the number of messages dropped, the
> date of the oldest message dropped and the date of the most recent
> message dropped. This topic would be defined so that the event is only
> emitted when the GetMessages request comes. So, even if my
> implementation has dropped messages at several times in the past since
> the last Get Messages request, there will only be one "MessagesDropped"
> event for each GetMessages request (think of it as the Schroedinger's
> cat of notifications, only realized when there is an observation). Note
> that it doesn't matter if the messages were dropped because of space
> limitations or because the GetMessages request asks for less messages
> than are available in the queue. This is just a way to tell people that
> they've missed something.

I don't like the idea of defining a special topic like this - it seems to
be encroaching into one of the TC non-goals of not defining "any 'standard'
or 'built-in' notification situations, events or messages". I am also a
little concerned about what action applications could be expected to take
as a result of these notifications, since by the time you get notified the
data has already been lost.

Regards,

Matt


Matt Roberts
IBM WebSphere Messaging Design and Development
Hursley Park, England. (44)-1962-815444
matt.roberts@uk.ibm.com
MP 211 / DE3H22



                                                                           
             "Vambenepe,                                                   
             William N"                                                    
             <vbp@hp.com>                                               To 
                                       "wsn-oasis"                         
             05/02/2005 00:27          <wsn@lists.oasis-open.org>          
                                                                        cc 
                                                                           
                                                                   Subject 
                                       [wsn] Pull-type notification        
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           
                                                                           




Hi all,

Thanks Steve for the "pull-type" proposal and thanks Matt for the
excellent minutes on 2/2, they made me feel like I was in the room for
the "pull-type" discussion.

Since there was no final decision on the "pull-type" question and since
I am very interested in this issue (as the person who initially raised
it), I would like to send some comments on Steve's proposal and on the
discussion that took place.

I think we need to put in place a simple, self-contained and useful
mechanism in BaseNotif. Advanced features should be pushed to
specialized policy statements and/or extension specs. So I agree for
example with the group that a generic queueing mechanism is out of scope
for us but if/when someone comes up with it people should be able to use
it in conjunction with our "pull-type" support.

Specifically, I think we can simplify our task in BaseNotif by not
worrying about buffer size and buffer utilization. We have to recognize
that of course the providers of "pull-type" subscription will have
limits in how many messages they can store and they might have to drop
old ones. But advanced support for this (such as being able to advertise
quality of service on your persistence and/or exposing how much of your
buffer is currently used) is best left outside of BaseNotif.

As a result, I would like to modify Steve's proposal by

- Removing the MaxBufferSize and CurrentBufferSize properties
- Removing the BufferSize attribute in the subscription request

Basically, when I subscribe I know that not all messages will be
persisted for ever and that the notification producer will do its best
but doesn't guarantee anything. Anything beyond that is out of scope for
BaseNotif in my view.

In addition, I would like to make three modifications to the GetMessages
request proposed by Steve:

The first change might actually just be a clarification since I am not
sure how Steve intended it to work. If there are 100 messages in the
queue and I do a GetMessages(40) then I will receive the 40 most recent
messages and the entire queue will be cleared. Basically, every
GetMessages request clears the entire queue in addition to getting some
or all of the most recent messages.

The second change is to have the GetMessages request contain 2 optional
elements (or attributes): MaxMessages, as currently defined, but also
OldestTime. If MaxMessages is present then don't send me any more than
this many messages. If OldestTime is present then don't send me any
message older than this date. The two filters can exist on their own or
co-exist. Yes, they are transitive.

The last change is to create a special topic called "MessagesDropped".
The payload for this topic contains the number of messages dropped, the
date of the oldest message dropped and the date of the most recent
message dropped. This topic would be defined so that the event is only
emitted when the GetMessages request comes. So, even if my
implementation has dropped messages at several times in the past since
the last Get Messages request, there will only be one "MessagesDropped"
event for each GetMessages request (think of it as the Schroedinger's
cat of notifications, only realized when there is an observation). Note
that it doesn't matter if the messages were dropped because of space
limitations or because the GetMessages request asks for less messages
than are available in the queue. This is just a way to tell people that
they've missed something.

Now to the question that the group wasn't able to agree on by the end of
the meeting (overload the subscribe operation or create a new
operation). At the messaging level, there is no major difference
between:

(1)
<Subscribe>
  <ConsumerReference/>
  <TopicExpressionDialect/>
  <SubscriptionPolicy>
    <UsePull>true</UsePull>
  </SubscriptionPolicy>
</Subscribe>

and

(2)
<Subscribe>
  <ConsumerReference/>
  <TopicExpressionDialect/>
  <Delivery>http://.../pull</Delivery>
</Subscribe>

and

(3)
<SubscribeForPull>
  <ConsumerReference/>
  <TopicExpressionDialect/>
</SubscribeForPull>

So the difference is not really a messaging difference but rather a
metadata difference. Because based on which of the three formats above
we choose, the "can I subscribe for pull-type notification" question is
answered in a different piece of metadata.

With (1) and (2) it is describe using a TBD policy statement expressed
in a TBD policy framework retrieved through a TBD metadata retrieval
mechanism. With (3) it is described by the WSDL.

(1) is the most generic mechanism (there are many ways to qualify a
subscription and this is one of them). (2) is less general (there are
many ways to deliver a notification and this is one of them). (3) is
most specific.

I prefer (3) because this makes the simple pull-type feature available
out of the box in BaseNotif. Also, (3) doesn't prevent a mechanism like
the ones in (1) or (2) to be overlaid (to distinguish between different
types of push-type delivery and different types of pull-type delivery).
It just recognizes that there is a major differences between pull-type
and push-type delivery and that the semantic difference of a
subscription to one or the other is sufficient to warrant using the
premier metadata mechanism for Web services (WSDL) to denote it.

Regards,

William