group management, and dup elimination

[Jacques Durand <JDurand@fsw.fujitsu.com>]

Title: group management, and dup elimination

Here is a proposal which, I believe, should address all concerns heard this afternoon,
first and foremost from Sunil. (Close to what he proposed in fact, but with nicer handling of "corner cases" :)
It looks a bit detailed, but I believe is in fact very "implementable".

Jacques

-----------------------------------------------------------------------------------------------

Background:
Let us start with precise semantics: We had 3 possible (non -exclusive)
semantics for expiryTime:
(1) business semantics (how long a message still makes sense for the application)
(2) RMP management semantics (how long should we try to make available to the app
before giving-up)
(3) dup elim semantics: when "at most once" is required, a message should be subject
to dup elim as long as it has not expired.

So far, our spec mostly talks of (2).
Several of us believe it is none of RMP business to enforce (1).
As for (3), there is an obvious impact from (2), as by definition, a duplicate of
an expired message would never be delivered, even if we skip the dup check.
That mostly spares us a dup check.

General approach: we endorse semantics (2) and (3).
For individual messages (group singletons), dup check is done over all past
messages that have not expired yet, and we keep this mesg ID store up-to-date.
But for groups, we do not want to have to store each message expiry time,
along with message IDs (GroupId + seq number). That would undermine the
fast and space efficient check that groups are supposed to provide in general.

General Proposal:
- we keep everything of what has been "decided" (or formulated for decision) so far.
- we still assume each individual message of a group has expiryTime.
Like for singletons, a message in a group would not be delivered to app if it has expired
when received. However we do NOT memorize this expirtyTime after first check.
(of course, in case we need to keep all undelivered messages of a broken sequence on receiver,
we keep the entire messages and therefore their expiryTime. We decide to give-up
if the min of all expirtyTimes of waiting messages has expired.)
- for the sake of simplicity, the receiver RMP will keep the ID of all past
messages for an active group, whether expired or not.
That is, once they have been delivered to app, RMP will NOT try to
manage expiry time of past delivered messages, and will NOT discard from memory
the message IDs of those expired, like it would for singletons.
Duplicate checks would be made over ALL past messages for the group, whether expired or not.
There is no added cost in fact, as these IDs are remembered as seq num intervals.
Will even be faster, as not discarding means fewer intervals.
 

Refined Proposal when groups terminate:

NOTE: In the 3 following termination cases, "discarding" a group means removing
everything (state, msg IDs, message bodies), and not doing anymore
duplicate checks for these. Any late message from this group will be treated
as from a new group.

Termination (t1): when a group terminates because its GroupExpiryTime is over,
it is completely discarded from RMP (state, message Ids and payloads).
No duplicate check is done against that group ever. If a "late duplicate" arrives,
it would be automatically discarded due to its expiry time.

Termination (t2): when a group terminates because its MaxIdleTime is past,
its messages may not be expired yet. We do not keep expiryTime of each message
of the group, however we keep the max of all expiryTimes of messages received
so far. We use this max as the GroupExpiryTime, and wait for this time -if needed-
before discarding the group. We fall back then to termination (t1).
In case a new message arrives after maxIdleTime expires but before the group expires,
we need to define what to do: I suggest we process it and keep bumping up the max
of all expiryTimes if needed (so dup check still enabled for the group).

Termination (t3): when a group terminates because it receives a status="end" message.
Note that this only tells us "no greater seq number will be received after".
Like in (t2), its messages may not be expired yet. We treat it like in (t2): the max
of all expiryTimes tells us when the group should be discarded, and not subject
to duplicate check. The group may still have missing messages that are late. Subcases:
(3.1)If the group was complete, still use max of expiryTimes before discarding it.
(so dup checks still possible, but we know are useless after that.)
(3.2) If group has missing messages, wait for these to the extent allowed by
either GroupExpiryTime, or maxIdleTime  (whichever was specified), then fall back on
either (t1) or (t2) respectively.