Alastair,
My intent is to show
that the replay message is quite valid.
Case 1: A recovered
participant in “Prepared Success” state may send a “Replay” message to a
coordinator that is in “Preparing” or “None” state.
A participant could
have successfully prepared (commit decision had been logged), and yet may not be
able to successfully send the “Prepared” message, because of network failure of
some sort. Such a participant would likely, after multiple unsuccessful attempts
to send the “Prepared” message to the coordinator, give up and abort the
transaction; that is, it may send out a “Rollback” message to all its
resources.
PRF: That should not be permitted behaviour (and isn't,
as far as I can tell). I'm assuming "not successfully send" includes getting a
lower-layer failure - but that generally doesn't mean you are sure the message
didn't go, just you aren't sure that it did. If the participant is quite
sure the Prepared never left and could not possibly have been received, then one
can safely consider that the WriteDone never happened, and go to aborting. In
which case it was never modelled as in PreparedSuccess. But "network failure"
generally doesn't let you know what the far end saw - and the whole point of
commitment protocols is to sort out the muddles that can then
arise.
The participant could fail at this time,
and upon recovery, will notice that it had indeed successfully prepared, and
will send out a “Replay” message to the coordinator to find out the outcome. The
coordinator, which may be in “Preparing” or “None” state (if the coordinator had
aborted) would send a “Rollback” message back, since the coordinator never
entered the “Prepared Success” state.
PRF: if the coordinator is still in Preparing, it now
knows this participant is prepared, so it could safely take a commit decision
and attempt to enter PreparedSuccess and commit. That is the whole
question of this issue. It has to be explained why that is undesirable
behaviour.
Case 2: A recovered
participant in “Prepared Success” state may send a “Replay” message to a
coordinator that in “Prepared Success”, “Committing”, or “Aborting”
state.
A participant that
has successfully prepared (commit decision had been logged), successfully sends
out a “Prepared” message to its coordinator, and then crashes. The coordinator
upon receipt of the “Prepared” message moves to the “Prepared Success” state.
The participant upon recovery will notice that it had indeed successfully
prepared, and will send out a “Replay” message to the coordinator to find out
the outcome. The coordinator, if in the “Prepared Success” state, would ignore
this message; and the participant would resend the replay message. The
coordinator when it makes a “Commit or Rollback decision” would move to a
“Committing” or “Aborting” state. When the replay message arrives at the
coordinator that is in a “Committing” or “Aborting” state, it would send a
“Commit” or “Rollback” message to the recovered participant, depending on the
final outcome.
Analysis:
It is only
appropriate for a recovered participant to send a “Replay” message to find out
the final outcome; resending a “Prepared” message may result in an erroneous
situation. For example, in case 1 above, the recovered participant should not
send a “Prepared” message to the coordinator, since it may have already aborted
the transaction locally. Hence, the “Replay” message has a role and a place in
the protocol.
PRF: I'm afraid you haven't shown that. If the
heuristically aborted participant of case 1 sends replay to the PreparedSuccess
coordinator of case 2, it will get told to commit, which it can no longer do.
Which is why heuristics are dangerous of course - and the agreement of this TC
is that heuristics will not be reflected in the protocol (they cannot be
absolutely forbidden in the real resources). So in case 1, if the Prepared might
possibly have been received, the participant must carry on as if it had - and on
receiving Commit will respond with Committed (and presumably raise local
management signals, outside our scope, to get someone to sort out the
problem)
Peter
From: Alastair Green
[mailto:alastair.green@choreology.com]
Sent: Wednesday, May 24, 2006 4:24
AM
To: Ram
Jeyaraman
Cc: Mark Little;
Peter Furniss; ws-tx@lists.oasis-open.org
Subject: Re: [ws-tx] Issue 052 - WS-AT:
Replay message generates protocol errors
Ram,
Thanks for your comments. I think they help
to clarify why Replay should go.
The "resend" definition of Replay is
perfectly acceptable in principle: it would be better worded as "interpret the
message as a resend of the last protocol message received by the Coordinator
View". It means: Replay is an algebraic quantity, to be valued by the
Coordinator View, given its state, as equivalent to the last message received
from the Participant.
If the Coordinator View is Active, then Replay can
be interpreted as "the last received message was Register, I must replay my
Register Response".
[In practice, using Replay to mean Register is going
to cause an almighty tangle. Because of the decision to use request-response for
WS-C messages, and to use [source endpoint] for notification messages like
Replay, it is not possible to make Replay equal Register. There is a related
issue as to which EPR a Replay-as-Register should be sent to: is it the
Coordinator View EPR, or the Registration Service EPR?. I think we should stop
this happening altogether.]
If the Coordinator View is Preparing
(awaiting a Prepared), then Replay can safely be interpreted as "the last
received message was Register; I must replay my Prepare
message".
In neither of these cases
is there any reason to abort the transaction. The conversation can
simply be resumed. That is issue 052.
If the Coordinator View is Prepared
Success, then Replay can safely be interpreted as "the last message received was
Prepared: I must send the outcome", irrespective of the state of CV.
Now
let us consider how these cases can arise.
There are three circumstances
where some kind of replay or resend may be needed or appropriate:
1.
Known communication failures (comms time out)
2. Unknown failures of the
interlocutor (it has crashed, its last message has not been delivered, it is so
slow as to appear to have failed)
3. Known failures of oneself (crash
recovery from a persistent record).
The correct reaction to
known-to-sender comms failures is to permit retry (according to any retry
strategy that the implementer finds useful). There is no need to say more: but
we do need to make that statement, and make it completely general, for all
messages. Not mandate, but permit.
Unknown failures manifest themselves
as "slow progress" -- e.g. CV expected an answer to Prepare but hasn't received
one.
The sender is again free to retry at will. The only thing we
know for certain is that an implementation that never retries cannot guarantee to
successfully complete the protocol: it depends on retries to achieve guaranteed
delivery. There are well-known adaptive retry strategies to avoid network
swamping, and we could do with a message (at least in BA) that warns that slow
progress is to be expected, to help refine such strategies. But the strategy
chosen is an implementation issue.
Again, we need the same general
statement: that any message may be resent at any time for any reason, with the
qualification that complete failure to attempt retries will produce a broken
implementation. I have already raised an issue that we should make such a
statement, and remove the comms time out line in the tables as redundant/overly
narrow.
The state tables are designed to cope with such duplications, and
will also cope with transport-induced duplications.
Known self-failure
(crash and recover) will only occur when the party concerned has logged its
state. It is an unspoken assumption (perhaps it needs, once again, to be spoken)
that an AT Participant will only
log when the state tables say it must log, i.e. when it goes prepared. If
we follow that assumption, then we don't need to worry about Replay arising when
the Coordinator is Active: it can only occur when the Participant has Prepared
Success (written its log), and that can't happen till the CV has moved to
Preparing (has sent Prepare to P).
[This would argue for making Replay
in Active N/A, incidentally. I think at present there is nothing to stop a P
logging early (e.g. before or after registering): if it did that then we are
wandering into the world of replaying Registers (which I think we should debate
and clarify).]
If the Participant has moved to Aborting or Committing
(volatile state) then recovery will lead it to revert to the logged state
Prepared Success. If the Participant has moved to None then it will not recover
(or at least, if it does, then it has retained a durable memory of the outcome,
and has no need to Replay). Therefore, we can concentrate solely on the case of
recovery of P in Prepared Success.
Recovering in the Prepared Success state, P sends
Replay, and always means by that:
"Prepared".
The CV, on receiving that message, is either
Preparing (i.e. awaiting the decision of the hidden overall coordinator C that
you don't want us to define), or is Prepared Success (the hidden overall
coordinator has decided to log its decision, but has not yet written the log),
or is Committing/Aborting (has already sent the Commit or Rollback message), or
is None.
The CV row Prepared defines the appropriate reaction of the CV
in all of these cases (plus or minus separate, orthogonal arguments about how to
react in None).
Therefore:
1) There is no reason for the CV to
react differently than it would if it received Prepared, in any of these states.
On reception Replay equals the semantic "prepared", is exactly like the message
Prepared.
2) There is no reason for comms time out to induce sending
Prepared, but crash recovery to induce sending Replay.
3) P can always
replay Prepared (instead of sending Replay). Replay is a redundant mechanism for
communicating the semantic "prepared".
4) If you want to replay Register,
the registering service can replay Register.
5) There is no reason to have Replay: it
is simply a synonym for Prepared (and possibly, maybe, if you really want to do
this, for Register). Issue 053.
Redefining the unnecessary seems ...
unnecessary.
Alastair
Ram Jeyaraman wrote:
Alastair,
Sorry for the delayed
response.
Part of the problem
here is the current definition of Replay: “resend the last appropriate protocol
notification”. The replay message does not necessarily indicate that the
participant has successfully prepared; it is possible that the participant is
still in a preparing state. Hence, the CV state table states that the
coordinator should send a rollback and move to Aborting state; that is, the
coordinator does not really know the possible conditions in which a participant
may send such a message.
If the replay message
were to be redefined, so it offers the guarantee that the participant had indeed
successfully prepared, the coordinator may be able to treat it differently; that
is, send the transaction outcome.
Ram,
I very much appreciate your responsiveness,
but I'm afraid that it seems you are arguing simply by assertion (that which is,
shall be).
1) Why should the coordinator abort in some circumstances
purely because of participant crash and recovery, despite the fact that the
participant has recovered in an identical state? That is the subject of this
issue 052. No-one has attempted to justify this behaviour thus far. Please can someone explain why this should be
so?
2) Anything that leaves the participant in doubt as to the
outcome requires it to communicate (again) with the C. If this occurs as a
result of detecting comms failure this leads to resend of Prepared. In undefined
circumstances, we are supposed to send Replay. The spec is entirely silent on
which circumstances, incidentally: it merely says that if it is received then it
is after "after recoverable failure". If we resolve that aggressive abort is
wrong (unjustified) then there will be no reason for having two messages.
Can someone please provide an argument for having two messages, if they
carry identical effective semantics?
3) There is nothing in
the spec to prevent replaying of Prepared. Where is this prohibited, defined,
circumscribed etc? Why should it be? Is it dangerous or
slow?
Alastair
Ram Jeyaraman wrote:
Alastair,
As we discussed
below, the replay message is typically sent by a participant that is in an
in-doubt situation. It should not be used for replaying a previous protocol
message as the specification currently
states.
The definition of
Replay message should read along these
lines:
“Upon
receipt of this notification, the coordinator may assume the participant has
suffered a recoverable failure. It should resend the transaction outcome (commit
or rollback protocol notification) to the in-doubt
participant.”
Ram,
Absolutely. So if Replay is pure synonym for
a resent Prepared (i.e. carries no additional semantic relevant to the outcome,
e.g. does not imply that P is aborting) then there is no reason for C to abort
the transaction on receiving the message. Put another way, there is no reason
for Replay to induce a different behaviour from a resent Prepared (which would
not cause abortion).
If Replay/Preparing is corrected to become identical
to Prepared/Preparing, then the unnecessary abort problem goes away (this
issue).
If the two rows become identical, then there is no need to have
a separate Replay message (it is redundant) -- the related
issue.
Alastair
Ram
Jeyaraman wrote:
Alastair,
A participant that has successfully prepared when it sends out a replay
(after a crash) genuinely wants to know what the outcome is, so it can
complete the in-doubt transaction during its recovery.
-----Original Message-----
From: Alastair Green [mailto:alastair.green@choreology.com]
Sent: Wednesday, May 10, 2006 4:15 AM
To: Mark Little
Cc: Peter Furniss; Ram Jeyaraman; ws-tx@lists.oasis-open.org
Subject: Re: [ws-tx] Issue 052 - WS-AT: Replay message generates
protocol errors
Ram, Mark --
I've got a quite a few thoughts on this, but I want to check with the TC
on a couple of premises, in case I am misunderstanding some unwritten
piece of design intent.
1. The text at l.221 of the spec defines the Replay message thus:
"Upon receipt of this notification, the coordinator may assume the
participant has suffered a recoverable failure. It should resend the
last appropriate protocol notification."
Does a Replay message for a Participant that crashed in the Prepared
Success and then recovered, carry the semantic:
a) "Have recovered, am in good state to proceed, i.e. am still
prepared", or
b) "Have recovered, was prepared, but am now aborting", or
c) "Have recovered, and may be prepared successfully, or may be
aborting", or
d) some other semantic, that I haven't thought of?
2. Is a Participant which crashed in the Prepared Success state, has
recovered from a failure and is still prepared (i.e. is in the same
state as it was prior to crash recovery) allowed to re-send Prepared? Or
better, can its decision to do so damage the consistency of the
transaction outcome, or slow down arriving at the outcome decision?
Alastair
Mark Little wrote:
Peter Furniss wrote:
I think it is likely the state table is being misinterpreted. I'm not
sure by who :-)
If you treat the state as referring to just one participant, you
either
get some very convoluted definitions of the internal events (c.f.
issue
048 - but more convoluted that the ones proposed there) or you
violate
atomicity.
We already agreed prior to the last f2f (in telecons) and at the last
f2f (during the meeting) that the state table is not referring to just
one participant.
Receiving a 'Prepared' message doesn't move the state to
PreparedSuccess
- that's done by "Commit Decision", and until then 'Replay' would
cause
an abort. You could define "Commit Decision" as meaning "receipt of
ok
vote for just this one participant", and take the state for this
participant to PreparedSuccess. But the only way to leave
PreparedSuccess is from "WriteDone" or "WriteFailed". Since a
'Aborted'
from another participant should certainly cause this participant to
be
rolled back, that 'Aborted' will have to trigger "WriteFailed", which
is
not an obvious interpretation.
But I think this issue, with 053 (eliminate Replay) is more about
whether Replay need ever force an abort. We may be looking at a
carry-over from connection-centric protocols, where it made sense to
force an abort if the connection broke before commit-time. In those
worlds (more or less all transaction protocols that weren't using xml
and/or web-services, I think), receipt of a recovery message before
the
connection was observed to break could only mean the connection break
was about to happen. But with WS-AT (especially because we have said
all
messages go on the underlying request) there is no connection to be
monitored anyway. The coordinator hasn't noticed that participant was
out of communication for a while, and now the participant says it is
ready for the commit. Why *require* the coordinator to abort ?
Agreed.
Of course that's not to say the coordinator cannot *choose* to abort
by
implementation option if replay is received (or any other
circumstance
that leads the coordinator to suspect a failure somewhere). It can
always do that if it hasn't progressed too far - it would appear in
the
tables as a User Rollback or Write Failed.
Yes, I'd like to see this as an implementation specific choice.
Mark.
Peter
-----Original Message-----
From: Ram Jeyaraman [mailto:Ram.Jeyaraman@microsoft.com] Sent: 06 May
2006 02:09
To: ws-tx@lists.oasis-open.org
Subject: RE: [ws-tx] Issue 052 - WS-AT: Replay message generates
protocol errors
Section 10 (AT specification) states "These tables present the view
of a
coordinator or participant with respect to a single partner". Thus,
the
coordinator states correspond to interactions with a single
participant.
The receipt of a participant vote "PreparedSuccess" triggers the
coordinator state to "PreparedSuccess" with respect to that
particular
participant, even though the coordinator may not have completed the
prepare phase for the rest of the participants.
Is it possible that the state table is likely being misinterpreted?
-----Original Message-----
From: Ram Jeyaraman [mailto:Ram.Jeyaraman@microsoft.com]
Sent: Thursday, April 06, 2006 10:50 AM
To: ws-tx@lists.oasis-open.org
Subject: [ws-tx] Issue 052 - WS-AT: Replay message generates protocol
errors
This is identified as WS-TX issue 052.
Please ensure follow-ups have a subject line starting "Issue 052 -
WS-AT: Replay message generates protocol errors ".
-----Original Message-----
From: Alastair Green [mailto:alastair.green@choreology.com]
Sent: Wednesday, April 05, 2006 5:07 PM
To: ws-tx@lists.oasis-open.org
Subject: [ws-tx] New Issue: WS-AT: Replay message generates protocol
errors
Issue name -- WS-AT: Replay message generates protocol errors
PLEASE DO NOT REPLY TO THIS EMAIL OR START A DISCUSSISON THREAD UNTIL
THE ISSUE IS ASSIGNED A NUMBER.
The issues coordinators will notify the list when that has occurred.
Target document and draft:
Protocol: WS-AT
Artifact: spec
Draft:
WS-AT CD 0.1 uploaded
Link to the document referenced:
http://www.oasis-open.org/apps/org/workgroup/ws-tx/download.php/17325/ws
tx-wsat-1.1-spec-cd-01.pdf
Section and PDF line number:
Coordinator View State Table, after l. 503
Issue type:
Design
Related issues:
New issue: WS-AT: Eliminate Replay message. New issue: WS-AT: Is
logging mandatory?
Issue Description:
Replay reactions defined in current CV state table will cause
unnecessary transaction aborts.
Issue Details:
The cells in row (Inbound Messages) Replay, columns (States) Active
and Preparing read:
Active: Send Rollback --> Aborting
Preparing: Send Rollback --> Aborting
Replay message means: "play it again Sam", not "demolish the piano".
Case A. If the last thing they sent was Prepared, and it got through
(we're Preparing and we've recorded their vote), and they've
recovered, and they're waiting for a Commit or a Rollback, then we
need to Ignore the Replay (just like if they send it when we've done
our own housekeeping, and moved to Prepared Success).
Case B. If the message didn't get through, and we've processed User
Commit then we could be in the Preparing state, but have no record of
their vote. In that case we'd have to replay Prepare to indicate to
them, send us your vote again.
Case C. If the last thing we received was Register, and we haven't
processed User Commit, then we're still Active and they won't have
logged. Replay won't happen on crash recovery (no log record to
recover off), but it could be used to say to the coordinator "Are you
still there? Should I crap out?" (i.e., because of impatience). We
can't stop them using Replay in that fashion. Our only sensible
response would have
to be: silence (we don't have a blank ack to a ping), i.e. to Ignore.
There is no harm in them doing this, even though it is pointless. You
could argue that this should be a N/A but that seems heavy-handed.
Proposed Resolution:
As the state tables do not differentiate between Preparing/no vote
recorded and Preparing/vote recorded, it seems easiest to always
resend Prepare in the Preparing state. Therefore:
Replace the current text in the cells in row (Inbound Messages)
Replay, columns (States) Active and Preparing with:
Active: Ignore --> Active
Preparing: Resend Prepare --> Preparing