Rel 50, 52, 57 revisited

[Jacques Durand <JDurand@fsw.fujitsu.com>]

Title: Rel 50, 52, 57 revisited

All:

Rel 50, 52, 57 adjusted to the proposed Ack semantics
(Ack generated only when RMP is successfully transfering the "responsibility"
of message processing to the upper layer).
Few changes actually, mostly:
- no longer talks of fault notifications,
- some hints (Rel 57) about efficient sender behavior to avoid useless resendings
in case an out-of-order sequence delays Acks (Sunil may have more precise suggestions)
- see also Note 1 in Rel 57 (Sunil, I did not actually take your suggestion here...)

Back to the notion of "Application" vs. RMP, Hamid BenMalek rightly observes
that several of our Figures in spec assume an "Application layer" on top
of the RMP, and the link between these is of course the implicit contract that
is assumed here between these two layers, the "app layer" obviously including
any intermediate device (queue, database...) that may facilitate communication
with a real business app. We just need to clarify this boundary.
 
Regards,

Jacques

 

Rel 50: (solution c) Semantics of ExpiryTime:
-------------------------------

Proposal:

ExpiryTime indicates the ultimate date after which the receiver RMP MUST
not deliver a received message to the application (delivery being
abstractly defined as the transfer of responsibility for processing a message, 
from RMP to the next layer.)

After a message has been sent for the first time, ExpiryTime in a message 
MUST NOT be modified in any case by the Sender, when resending the message:
two messages with same ID (duplicates) MUST also have same ExpiryTime.

When a message expires on the Sender side before being sucessfully sent,
a Sender RMP MUST NOT send it or resend it, and must communicate a delivery failure 
to the Sender application.

NOTES: 

- Given the above definition of ExpiryTime, in case duplicate elimination is required, 
when a received message is processed, it is sufficient to only check for its duplicates 
among IDs of past messages that have not expired yet at the time of the duplicate check. 




Rel 52: (Solution c)(New rules for persistence, also, related "timing issues") 
-------

Proposal:

Q1: When can a recipient "forget" about a sequence? 
How long must an out-of-order message be stored prior to giving up on 
ever delivering it to the application?

Answer: see Rel57.

Q2: How do these time windows relate to the TTL (expiration) of an individual message?

Answer: see Rel50 and Rel57.

Q3: Within what window will a duplicate be detected? 

[Short Answer:] duplicates will be detected over all non-expired messages. 
Since by definition, messages that are expired before the RMP processing is over 
will not be made available to the application, and because ExpiryTime is identical across
duplicates, there is no risk ever to deliver two duplicates to the receiver application, 
when restricting the duplicate check over the IDs of non-expired messages. 

[Long answer]: 

General rule for persisting the ID of a message:
---------------------------------------------

The duplicate elimination feature requires persisting the message ID
(GroupId and optionally the Sequence Number), after the message
has been processed and delivered (made available) to the application.
The general rule about message ID persistence, is that an RMP
MUST persist the ID of a message at least until the message expires
regardless whether the message has been delivered or not to the application.

It is RECOMMENDED that an RMP removes the ID of a singleton message 
(message with GroupId but no SequenceNumber) from its persistent store after it expires.
However, unlike for singletons, it is RECOMMENDED to NOT remove the ID of a message that
belongs to a non-singleton group, when the message expires. The reason is, message IDs
for groups can be stored efficiently as intervals of integers.

NOTE: For groups, duplicate checks may be made over all past messages of the group, 
whether expired or not, without loss of performance. The check will in fact be faster, 
because not discarding expired IDs means fewer intervals to look up.

General rules for terminating a group, and for removing the state of a group:
-----------------------------------------------------------------

NOTE: these termination rules apply to both ordered and unordered groups.

NOTE: In all termination cases (t1, t2, t3, t4) below, it is not necessary to remember 
the ExpiryTime of all messages of a group, as only the maximum of all ExpiryTime values 
is needed.

NOTE:Termination of a group, and removal of its state are two distinct events.
- When a group terminates, both in sender and receiver, no message is expected to be
sent or received for this group anymore. If a message is received after the
termination of a group, with same GroupId as the terminated group, it is considered
by the Receiver as belonging to a new group (the Receiver is not required
to verify that a new GroupId value has not already been used in a previous group, 
as this test is impractical).
- When the state of a group is removed, any trace of the group (including status, groupId, 
current sequence number, as well as all received message IDs for the group 
e.g. SequenceNumber intervals) are removed, and therefore not available anymore for 
duplicate checks. State removal may occur either at termination time or later,
depending on the following rules. 

Termination (t1): 
Triggering event: GroupExpiryTime is over.
Receiver side: The RMP MUST NOT accept any new message for this group and MUST
terminate the group. It is RECOMMENDED that its state be removed as soon as possible.
No duplicate check needs to be done against that group ever. If a "late duplicate" arrives,
it would never be delivered to the application, as its ExpiryTime, which is always earlier 
than GroupExpiryTime, would have expired.
Sender side: The group MUST be terminated, and its state removed from the RMP.

Termination (t2): 
Triggering event: GroupMaxIdleTime is over.
Receiver side: Except for an ordered group that is out-of-order (see this case in Rel52), 
the group MUST be terminated. But unlike (t1), some of its 
past messages may not have expired yet, and therefore their ID still be needed 
for duplicate checks. If we define max(all ExpiryTime) as the max of all ExpiryTimes 
of messages already received for a group, an RMP MUST persist the state of a group
even after termination of the group, at least until max(all ExpiryTime), in case 
duplicate elimination is required.
Sender side: The group MUST be immediately terminated, and its state is removed from the RMP
when either (1) the time elapsed since the last sent message (including retries) exceeds 
GroupMaxIdleTime, or (2) for an ordered group, a message has failed to be successfully 
delivered (as described in Rel 57), whichever occurs first.

Termination (t3): 
Triggering event: the RMP receives a status="end" message. The group had either
GroupExpiryTime or GroupMaxIdleTime specified.
Note that such an event only tells that "no greater seq number will ever be received after", 
but late messages may still arrive for this group. 
Subcase t3.1: The group was complete on receiver side. 
Receiver side: The group MUST be terminated. 
However, its state is removed according to (t1) or (t2), depending which termination
criterion was given.
Sender side: The group is also known to be complete, and MUST be terminated. 
However, if guaranteed delivery was required, all sent messages must either have been
acknowledged, or been faulted, before termination. 
Subcase t3.2: The group was not complete on receiver side. 
Then the receiver RMP and sender RMP MUST apply rules of (t1) or (t2), depending which one 
of GroupExpiryTime or GroupMaxIdleTime was specified.

Termination (t4): 
Triggering event: the RMP receives a status="end" message. The group had neither
GroupExpiryTime nor GroupMaxIdleTime specified.
Subcase t4.1: The group was complete on receiver side. 
Receiver side: The group MUST be terminated. The date of removal of its state 
is based on rules for (t2).
Sender side: see subcase t3.1 above.
Subcase t4.2: The group was not complete on receiver side.
This event, by itself, does not cause the termination of the group. 


Rel 57: (Solution c) Ordering and Missing Message Behavior
---------------------------------------

Proposal:

[short answer]: 
the RMP will wait until expiration of the earliest ExpiryTime of the 
out-of-order messages. All subsequent out-of-order messages will then be discarded.
However, a shortage in storage resources may lead to discard out-of-order 
messages before the group terminates.  

[long answer]:
Given the definition of Guaranteed Ordered Delivery, a message must not be delivered
to the receiver application if not all previous messages were.
Therefore in case a Receiver RMP decides to not wait any longer for a missing
message, the already received out-of-order messages for the group MUST not be delivered 
to the application, and can be discarded. 

[NOTE 1: we could have discarded only the "subsequent" messages - i.e. with greater 
seq numbers than the one expired. Indeed, "previous" out-of-order messages still have
a chance to be delivered to the app when the missing message arrives,
but the group is broken anyway, and future messages will not be accepted. 
In addition, the fact that a pending message expires makes it likely that the 
missing message also expires soon. So there is little benefit in keeping the prior
messages to the expired one. In addition, that makes the RMP implementation more complex.]

On the Receiver side, as soon as the earliest ExpiryTime of pending (stored) 
out-of-order messages has expired, all the pending and future out-of-order messages, 
MUST not be delivered to the application. 
The RMP MUST terminate the out-of-order group as early as convenient after this date, 
and out-of-order messages must be discarded. 
(see Rel 52 for definition of group temination)

When sending messages for an ordered group, the Receiver MAY decide to NOT resend a 
non-acknowledged message before all the previous messages with smaller sequence numbers 
have been acknowledged. It MAY also decide to resend a non-acknowledged message
only when the total number of non-acknowledged messages is below a threshold that
is implementation dependent.

On the Sender side, the RMP MUST terminate an ordered group as early as possible after
one of these events occurs, whichever comes first:
(1) a message for the group has not been acknowledged after the last retry plus a 
buffer time (e.g. retryinterval).
(2)the ExpiryTime of a message to be sent or resent for the group, has expired before 
the message has been acknowledged.
Deciding that a message has "Not been acknowledged" must require a failed poll request 
from the Sender, in case the reply pattern is "poll", or must exhaust the expected
delay in receiving a multiple-acknowledgement.

NOTE 2: discarding of out-of-order messages of a group, and more generally terminating a 
group, does not mean yet the removal of the group state, as message IDs may still 
be needed for duplicate checks.