Re: [ws-rx] AS need for ordered delivery?

[Alastair Green <alastair.green@choreology.com>]

Duane et al,

I am trying to get my head around the ramifications of this discussion.

The delivery assurance patterns in the first instance concern the retry 
responsibilities of the RM endpoints with respect to at least once 
delivery. Does the sender resend until acknowledged, or does it not? 
Does the receiver acknowledge, or does it not? If the protocol is always 
at-least-once (does not have a mode where delivery drops are tolerated) 
then there is nothing to say on the wire, because the at-least-once 
contract is implicit.

Exactly-once delivery, and ordered delivery, are contracts between the 
receiving endpoints and the consuming application, but have differing 
requirements upon the RM-RM protocol per se.

EO delivery requires that the RMD receive at least once from the RMS, 
and deliver exactly once to the AD (eliminate dups). This implies no 
addition to the ALO-capable inter-RM protocol.

Ordered delivery requires that the protocol communicate sequence ids and 
an end-of-sequence marker (or a logical equivalent). These ids can then 
be used by the RMD to ensure ordered delivery.

In each of these cases the consuming app[s] in the widest sense (the 
application[s] using the stream of inbound messages, which could include 
administrative tools to enforce application behaviours on individual 
consuming apps) could (either by API or configuration) demand of a 
suitably smart  RMD that it observe an EO delivery contract, or an 
ordered delivery contract. They could also demand the combination: that 
the contract is ordered-EO. Other than the presence of sequence 
apparatus on the wire, there is no implication on the inter-RM contract. 
These are all receiver-side behaviours.

There are several stripes of a possible higher level of behaviour of the 
whole system, however, which would effectively be concerned with 
inter-app contracts, communicated by the inter-RM protocol, and stated 
by the sending app to the RMS:

1) That the message must not be accepted unless it can and will be 
delivered to the AD only EO

2) That a sequence must  not be accepted unless it  can and will be 
delivered monotonically to the AD

3) That a sequence must not be accepted unless it can and will be 
delivered monotonically and EO, i.e. without repetition, to the AD

4) That a sequence must not be accepted unless it can and will be 
delivered only as whole, and not in part.

In my view this additional level of contract is questionable, but it's 
not completely obvious that it shouldn't exist. It could be handled 
quite differently, by an out-of-band QoS declaration on the part of the 
receiver, although that makes life quite complicated, in terms of static 
config on a per-message type basis.

Certain distributed applications depend for their overall robustness on 
the use of EO semantics. Attempts to create stratified transactions (to 
use Frank Leymann's term), will fail without EO. However, EO delivery is 
only one part of the inter-app contract (it is also required that 
processing is made EO by transactionally binding processing to the 
reception of the message, or by guarantees of idempotent processing). Is 
there any case where EO delivery does not imply and is not meant to 
imply such EO processing guarantees? If so, then the communication of EO 
as an intent or mandate is not sufficient to usefully express the whole 
inter-app contract. If EO delivery can only be taken usefully to mean EO 
processing then the EO contract statement may be very useful.

The issue of sequences could be thought of differently, and more 
completely, by a hierarchical view of messages: messages A and B are 
dependent upon, children of message M, which in turn could be dependent 
on message X. This would imply that acknowledgements would logically 
apply at each level of the hierarchy (in practice, the last child ack 
would be an ack of the parent). Such a  hierarchy would  enable  
statement of the intent: receive and deliver a whole unit (in effect 
permitting the fragmentation of large messages to permit streaming 
delivery).

(Parenthetically, do we mean by sequencing: simply deliver in order, or 
do we mean: deliver a bounded set in order? The latter requires a 
logical end-of-sequence, the former merely sequence ids.)

The danger, in defining an interoperable protocol, is that all of these 
behaviours and issues get eliminated in pursuit of the laudable goal of 
conciseness, but in fact implementations constantly extend the protocol 
to allow such inter-app contracts to be communicated -- and it then 
ceases to be interoperable in practice.

There are two routes to resolving this, as I see it.

One is to rigourously examine and triage use cases and requirements 
(perhaps best based on existing proprietary RM vendors stating what they 
and their customers actually do in practice, as this is not greenfield 
technology, to say the least).

The second route is  to defer all such issues to a future spec in the 
composable architecture, concerned with QoS or policy. The danger in the 
second approach is that such a policy spec is never written, isn't on 
the right roadmaps etc. And if it is written, it will likely require 
space for policy declarations anyway in the wire protocol.

This last point encourages the thought that there should be (at least) 
some well defined space for inter-app policy declarations, even if the 
vocabulary of policy assertions is not fully defined, and some means for 
faulting in the face of an unprocessable policy. The extent of 
definition of the vocabulary would then become a distinct discussion.

Alastair



Duane Nickull wrote:
> "There is a more relaxed and performant view, which allows streaming 
> delivery: this says: nothing must be delivered to the application that 
> is out of order (or a duplicate in case of EO delivery), even if not all
>
> of the sequence has yet been received."
>
> Alistair:
>
> Is it reasonable for this TC to define a specific implementation
> scenario for "in order delivery" and state it is applicable behind
> service interfaces?  The opacity of service concept frees implementers
> of each service to make those decisions and if this TC tries to reach
> behind the service and mandate a specific implementation, I would
> object.
>
> Also - is it reasonable for an RMD to hold onto every single message
> until it has received the entire sequence?  Why can't RMD deliver
> message 1, 2 and 3 to the AD right away, then send a Nack back to the
> RMS if 4 is missing and 5 comes next.   In this case, it could hold back
> 5 until 4 is present, but given that 1 and 5 may be some considerable
> time apart this may itself be suboptimal.
>
> Question for all:
>
> Is it really the physical delivery or the ability to subsequently
> understand the ordering of the messages that matters?
>
> Duane
>
>
>
>