[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [wsbpel-implement] Fault tolerance considerations
Mike,
Interesting scenario. I have run into similar problems using HTTP in
a product I worked on a while back. Our solution was to periodically
send a "100 Continue" response to the client, thus keeping the
connection alive, and the client happily waiting. It was an okay
solution for the particular product, but in general it encourages a lot
of idle network resources to be tied up in open connections. It also
puts a crimp in the scaling story.
Doesn't SOAP 1.1 talk about "natural" bindings for the
request/response MEP, but does not mandate that request/response be
truly synchronous? (Just a vague recollection; I can't seem to be able
to raise the w3c site right now...).
I believe WS-Routing allows specification of a return path, and some
extra context information, so that one could easily correlate an
asynchronous response to the originator of the request. WS-Reliability
and ebXML MS have mechanisms for such message correlation as well. As
long as BPEL is built on the abstract WSDL message model, it can largely
ignore binding-specific issues. Of course, interoperability demands that
we at least consider them! If WS-I BP 1.0 is considered the best bet for
interoperability for BPEL implementations, then we should give such
HTTP-related issues extra attention.
Cheers,
-Ron=20
Marin, Mike wrote:
<!--[if !supportEmptyParas]--> <!--[endif]-->
Well, I have the same problem, and you do not need a crash to do that.
The problem is that BPEL prescribe receive-reply as implementing a
synchronous WSDL operation, when in practice you cannot enforce it. You
just need add a wait for a week between the receive and the reply, and
I'm sure you do not want to keep the connection open for that long.
<!--[if !supportEmptyParas]--> <!--[endif]-->
I opened issue 17 (Asynchronous operations) a while back, but have not
have time to pursue it. IMHO the receive / reply pair does requires an
asynchronous WSDL binding (one that does not require the connection to
remain open). In theory, you could define such a binding, but nobody
will be able to use it because first is not WS-I compliant, and second
does not fit most WSDL implementation frameworks.
<!--[if !supportEmptyParas]--> <!--[endif]-->
It may be that WS-Routing provides a solution to this issue by allowing
a reverse message path for the reply. But, I have not had time to study
this alternative.
<!--[if !supportEmptyParas]--> <!--[endif]-->
In any case, I'm also interested on see (read) how others are tackling
this implementation issue....
<!--[if !supportEmptyParas]--> <!--[endif]-->
--
Regards,
Mike Marin
<!--[if !supportEmptyParas]--> <!--[endif]-->
-----Original Message-----
From: Ron Ten-Hove [mailto:Ronald.Ten-Hove@Sun.COM]
Sent: Tuesday, October 14, 2003 4:25 PM
To: bpel implementation
Subject: [wsbpel-implement] Fault tolerance considerations
<!--[if !supportEmptyParas]--> <!--[endif]-->
Folks,
I was recently given an interesting question from one of my
development teams, and I thought it would be of interest to this group,
since it touches on universal implementation issues.
The question is based on the following scenario: given a process
something like this:
<sequence>
<receive name=3D"rcv" ... />
<assign name=3D"as1" ... />
<invoke name=3D"inv" ... />
<assign name=3D"as2" ... />
<reply name=3D"rep" ... />
</sequence>
=20
The <receive> and <reply> activities are part of a request-response MEP,
bound to SOAP, so that the request-response is synchronous (uses the
same connection for request and response).
Simple enough. But suppose that during execution of an instance of
the above process, somewhere after the <receive> activity is completed
but before the <reply> activity is done, the BPEL engine suffers a
crash. Since we have the full state persistence, recovery is simple
enough. We can therefore finish creating the reply, but this is rather
useless, since the client connection is lost.=20
So what is the right thing to do under these circumstances? Should
the engine, upon recovery in this situation, fault the running activity?
Should it continue to the reply activity, and presumably fault because
the connection is closed?
What of the client program? It sees that the HTTP connection closed
while awaiting a response to the request. It might reasonably resend the
request (HTTP being what it is). If this is the expected behaviour,
might it not be appropriate for the BPEL engine offering the service our
client is using to, upon recover, "roll back" or otherwise compensate
the completed activities in the sequence (not shown in the process
above), to the point of the <receive> activity, and restart the receive?
I know that some of these complexities are the result of using
unreliable messaging, and you get what you pay for, right? On the other
hand, this illustrates some interesting states that a BPEL
implementation might have to deal with, which aren't discussed in the
specification. At the very least, we have some unspecified faults to
deal with -- presumably implementation specific.=20
So what are other implementers doing in this case? Generating a
fault of one sort of another, or performing more heroic efforts to
recover from the crash? I'm just interested in general approaches, since
we don't want to require NDAs here! My development team is busy trying
to create some recovery mechanisms for the scenario above, based on some
sort of client/server interaction (client retries being the most likely
sort). These guys are pretty clever, so I wouldn't doubt that they could
invent something that, in many cases, actually recover from the crash
scenario above.=20
Thoughts? Is anyone else concerned about crash recovery, perhaps
with different scenarios?
-Ron
------_=_NextPart_002_01C39321.F2323D41
Content-Type: text/html;
charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Dus-ascii">
<TITLE>Message</TITLE>
<META content=3D"MSHTML 6.00.2800.1264" name=3DGENERATOR></HEAD>
<BODY text=3D#000000 bgColor=3D#ffffff>
<DIV><SPAN class=3D141303613-15102003><FONT face=3DArial color=3D#0000ff =
size=3D2>Dear=20
Ron, Mike, Ugo, and others,</FONT></SPAN></DIV>
<DIV><SPAN class=3D141303613-15102003><FONT face=3DArial color=3D#0000ff =
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D141303613-15102003><FONT face=3DArial color=3D#0000ff =
size=3D2>Is=20
this problem not occurring because of layer violation. Surely BPEL =
- and=20
BPEL implementations - should not need to worry beyond setting a timeout =
at the=20
sending side on an invoke to which a reply is expected. At the =
BPEL should=20
just a hand a message over to the SOAP HTTP implementation, and receive =
messages=20
back from it. Exactly how the message is sent (and in particular =
whether=20
the response rides on an HTTP response or request is surely not a BPEL=20
concern (???)</FONT></SPAN></DIV>
<DIV> </DIV>
<DIV class=3DSection1>
<P style=3D"MARGIN: 6pt 0cm; tab-stops: 119.65pt 309.05pt" =
align=3Dleft><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Tahoma; =
mso-bidi-font-size: 12.0pt">Best=20
Regards,<?xml:namespace prefix =3D o ns =3D=20
"urn:schemas-microsoft-com:office:office" /><o:p></o:p></SPAN></P>
<P style=3D"MARGIN: 6pt 0cm; tab-stops: 119.65pt 309.05pt"><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Tahoma; =
mso-bidi-font-size: 12.0pt">Tony</SPAN><I><SPAN=20
style=3D"COLOR: navy; FONT-FAMILY: Tahoma"> <SPAN=20
style=3D"mso-tab-count: =
1"> &nbs=
p;  =
; =20
</SPAN></SPAN></I><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial"><o:p></o:p></SPAN></P>
<TABLE=20
style=3D"BORDER-RIGHT: medium none; BORDER-TOP: medium none; =
BORDER-LEFT: medium none; BORDER-BOTTOM: medium none; BORDER-COLLAPSE: =
collapse; mso-border-alt: solid windowtext .5pt; mso-padding-alt: 0cm =
5.4pt 0cm 5.4pt"=20
cellSpacing=3D0 cellPadding=3D0 border=3D1>
<TBODY>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 119.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop align=3Dleft width=3D160 rowSpan=3D5>
<P style=3D"MARGIN-TOP: 18pt"><I><SPAN=20
style=3D"COLOR: navy; FONT-FAMILY: Tahoma"><A=20
title=3D"The interplays of commerce" =
href=3D"http://www.choreology.com/";><IMG=20
height=3D81 src=3D"cid:141303613@15102003-0394"; width=3D145 =
border=3D0=20
v:shapes=3D"_x0000_i1025"></A><o:p></o:p></SPAN></I></P></TD>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 276.3pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D368 colSpan=3D2>
<P style=3D"MARGIN: 6pt 0cm"><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial">Tony=20
Fletcher<o:p></o:p></SPAN></P>
<P style=3D"MARGIN: 6pt 0cm"><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial">Technical=20
Advisor</SPAN><SPAN style=3D"COLOR: black; FONT-FAMILY: Verdana">=20
<BR></SPAN><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 7.5pt">Choreology=20
Ltd.<BR>68, Lombard Street, London EC3V 9L<FONT size=3D1> =
</FONT>J =20
UK</SPAN><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial"><o:p></o:p></SPAN></P></TD></TR>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 47.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D64>
<P class=3DMsoNormal><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: =
7.5pt">Phone: </SPAN><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 12.0pt">=20
<o:p></o:p></SPAN></P></TD>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 228.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D305>
<P><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 7.5pt">+44=20
(0) 870 7390076</SPAN></P></TD></TR>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 47.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D64>
<P class=3DMsoNormal><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: =
7.5pt">Mobile:</SPAN><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 12.0pt">=20
<o:p></o:p></SPAN></P></TD>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 228.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D305>
<P><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 7.5pt">+44=20
(0) 7801 948219</SPAN><I><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: =
12.0pt"><o:p></o:p></SPAN></I></P></TD></TR>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 47.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D64>
<P class=3DMsoNormal><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: =
7.5pt">Fax: </SPAN><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 12.0pt">=20
<o:p></o:p></SPAN></P></TD>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 228.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D305>
<P><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 7.5pt">+44=20
(0) 870 7390077</SPAN></P></TD></TR>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 47.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D64>
<P class=3DMsoNormal><SPAN=20
style=3D"FONT-SIZE: 8pt; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: =
12.0pt">Web:<o:p></o:p></SPAN></P></TD>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 228.65pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D305>
<P><I><SPAN=20
style=3D"FONT-SIZE: 8pt; COLOR: red; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 12.0pt"><A=20
title=3D"The interplays of commerce" =
href=3D"http://www.choreology.com/";><SPAN=20
style=3D"COLOR: red; FONT-STYLE: normal; mso-bidi-font-family: =
'Times New =
Roman'">www.choreology.com</SPAN></A><o:p></o:p></SPAN></I></P></TD></TR>=
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 395.95pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D528 colSpan=3D3>
<P><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: =
12.0pt">Cohesions™<o:p></o:p></SPAN></P></TD></TR>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 395.95pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D528 colSpan=3D3>
<P style=3D"MARGIN-BOTTOM: 6pt"><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 12.0pt">Business=20
transaction management software for application=20
coordination<o:p></o:p></SPAN></P></TD></TR>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 395.95pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D528 colSpan=3D3>
<P style=3D"MARGIN-BOTTOM: 6pt"><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: red; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 12.0pt">
<P></P>Work: tony.fletcher@choreology.com</SPAN>=20
<P></P></TD></TR>
<TR>
<TD=20
style=3D"BORDER-RIGHT: medium none; PADDING-RIGHT: 5.4pt; =
BORDER-TOP: medium none; PADDING-LEFT: 5.4pt; PADDING-BOTTOM: 0cm; =
BORDER-LEFT: medium none; WIDTH: 395.95pt; PADDING-TOP: 0cm; =
BORDER-BOTTOM: medium none"=20
vAlign=3Dtop width=3D528 colSpan=3D3>
<P style=3D"MARGIN-BOTTOM: 6pt"><SPAN=20
style=3D"FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: Verdana; =
mso-bidi-font-family: Arial; mso-bidi-font-size: 12.0pt">Home:=20
<A=20
=
href=3D"mailto:amfletcher@iee.org";>amfletcher@iee.org</A></SPAN></P></TD>=
</TR></TBODY></TABLE></DIV>
<BLOCKQUOTE dir=3Dltr style=3D"MARGIN-RIGHT: 0px">
<DIV></DIV>
<DIV class=3DOutlookMessageHeader lang=3Den-us dir=3Dltr =
align=3Dleft><FONT=20
face=3DTahoma size=3D2>-----Original Message-----<BR><B>From:</B> Ugo =
Corda=20
[mailto:UCorda@SeeBeyond.com] <BR><B>Sent:</B> 15 October 2003=20
02:41<BR><B>To:</B> Ron Ten-Hove; Marin, Mike<BR><B>Cc:</B> bpel=20
implementation<BR><B>Subject:</B> RE: [wsbpel-implement] Fault =
tolerance=20
considerations<BR><BR></FONT></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2>Ron,</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial size=3D2>From =
the point of=20
view of both SOAP 1.1 and WSDL 1.1 the question of whether we are =
dealing with=20
a synchronous request/response or an asynchronous one is up to the=20
binding.</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial size=3D2>From =
SOAP 1.1,=20
sec. 2:</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial =
size=3D2>"SOAP=20
implementations can be optimized to exploit the unique characteristics =
of=20
particular<BR>network systems. For example, the HTTP binding described =
in=20
section 6 provides for SOAP<BR>response messages to be delivered as =
HTTP=20
responses, using the same connection as the<BR>inbound=20
request".</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial size=3D2>As =
you mention=20
below, a synchronous binding to HTTP is just a way to "optimize" the=20
request/response MEP, and in principle other non-synchronous bindings =
are=20
possible.</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial =
size=3D2>Things are similar=20
from the point of view of WSDL 1.1. Sec. 2.4.2, Request-response=20
Operation, says:</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial =
size=3D2>"Note that a=20
request-response operation is an abstract notion; a particular binding =
must be=20
consulted to<BR>determine how the messages are actually sent: within a =
single=20
communication (such as a HTTP<BR>request/response), or as two =
independent=20
communications (such as two HTTP requests)".</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial size=3D2>So, =
if it was just=20
a matter of looking at SOAP and WSDL, I would say that BPEL should or =
should=20
not generate a fault upon resuming based on the particular binding=20
used.</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial size=3D2>But =
BPEL seems to=20
take a different direction when it comes to the receive/reply =
pattern, in=20
the sense that it seems to strongly imply that a receive/reply is only =
acceptable with a synchronous binding, and that receive/invoke, =
combined with=20
call back interfaces, should instead be used with =
asynchronous=20
bindings. The discussion in BPEL 1.1, page 23, second paragraph, seems =
to be=20
rather clear in this respect.</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial size=3D2>I =
think BPEL takes=20
this direction because of the desire of distinguishing =
synchronous=20
connections from asynchronous ones at the language level, while at the =
same=20
time restraining from specifying any bindings. I guess if WSDL allowed =
to=20
distinguish synchronous/asynchronous BPEL could have used that, but =
WSDL 1.1=20
does not and BPEL uses receive/reply and receive/invoke, plus the call =
back=20
interfaces, to distinguish the two cases.</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2></FONT></SPAN> </DIV>
<DIV><SPAN class=3D539531401-15102003><FONT face=3DArial=20
size=3D2>Ugo</FONT></SPAN></DIV>
<DIV><SPAN class=3D539531401-15102003></SPAN><SPAN=20
class=3D539531401-15102003></SPAN><FONT face=3DTahoma><FONT =
size=3D2><SPAN=20
class=3D539531401-15102003><FONT=20
face=3DArial></FONT></SPAN></FONT></FONT> </DIV>
<DIV><FONT face=3DTahoma><FONT size=3D2><SPAN=20
class=3D539531401-15102003> </SPAN>-----Original=20
Message-----<BR><B>From:</B> Ron Ten-Hove=20
[mailto:Ronald.Ten-Hove@Sun.COM]<BR><B>Sent:</B> Tuesday, October 14, =
2003=20
5:48 PM<BR><B>To:</B> Marin, Mike<BR><B>Cc:</B> bpel=20
implementation<BR><B>Subject:</B> Re: [wsbpel-implement] Fault =
tolerance=20
considerations<BR><BR></DIV></FONT></FONT>
<BLOCKQUOTE dir=3Dltr=20
style=3D"PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px =
solid; MARGIN-RIGHT: 0px">Mike,<BR><BR> =20
Interesting scenario. I have run into similar problems using HTTP in =
a=20
product I worked on a while back. Our solution was to periodically =
send a=20
"100 Continue" response to the client, thus keeping the connection =
alive,=20
and the client happily waiting. It was an okay solution for the =
particular=20
product, but in general it encourages a lot of idle network =
resources to be=20
tied up in open connections. It also puts a crimp in the scaling=20
story.<BR><BR> Doesn't SOAP 1.1 talk about =
"natural"=20
bindings for the request/response MEP, but does <B>not</B> mandate =
that=20
request/response be truly synchronous? (Just a vague recollection; I =
can't=20
seem to be able to raise the w3c site right=20
now...).<BR><BR> I believe WS-Routing allows =
specification=20
of a return path, and some extra context information, so that one =
could=20
easily correlate an asynchronous response to the originator of the =
request.=20
WS-Reliability and ebXML MS have mechanisms for such message =
correlation as=20
well. As long as BPEL is built on the abstract WSDL message model, =
it can=20
largely ignore binding-specific issues. Of course, interoperability =
demands=20
that we at least consider them! If WS-I BP 1.0 is considered the =
best bet=20
for interoperability for BPEL implementations, then we should give =
such=20
HTTP-related issues extra attention.<BR><BR>Cheers,<BR>-Ron =
<BR><BR>Marin,=20
Mike wrote:<BR>
<BLOCKQUOTE=20
=
cite=3Dmid69FB2CA668C6D841BE0326C13071C69F0E1B29@hq-ex2kpo1.filenet.fn.co=
m=20
type=3D"cite">
<META content=3DWord.Document name=3DProgId>
<META content=3D"Microsoft Word 9" name=3DGenerator>
<META content=3D"Microsoft Word 9" name=3DOriginator><LINK=20
href=3D"cid:filelist.xml@01C39277.7F638D20"; =
rel=3DFile-List><!--[if gte mso 9]><xml>
<o:OfficeDocumentSettings>
<o:DoNotRelyOnCSS/>
</o:OfficeDocumentSettings>
</xml><![endif]--><!--[if gte mso 9]><xml>
<w:WordDocument>
<w:Zoom>0</w:Zoom>
<w:DocumentKind>DocumentEmail</w:DocumentKind>
<w:EnvelopeVis/>
</w:WordDocument>
</xml><![endif]-->
<STYLE>@font-face {
font-family: Tahoma;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; =
mso-header-margin: .5in; mso-footer-margin: .5in; mso-paper-source: 0; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; COLOR: black; FONT-FAMILY: "Times =
New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; =
mso-fareast-font-family: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; COLOR: black; FONT-FAMILY: "Times =
New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; =
mso-fareast-font-family: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; COLOR: black; FONT-FAMILY: "Times =
New Roman"; mso-style-parent: ""; mso-pagination: widow-orphan; =
mso-fareast-font-family: "Times New Roman"
}
P.MsoAutoSig {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; COLOR: black; FONT-FAMILY: "Times =
New Roman"; mso-pagination: widow-orphan; mso-fareast-font-family: =
"Times New Roman"
}
LI.MsoAutoSig {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; COLOR: black; FONT-FAMILY: "Times =
New Roman"; mso-pagination: widow-orphan; mso-fareast-font-family: =
"Times New Roman"
}
DIV.MsoAutoSig {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; COLOR: black; FONT-FAMILY: "Times =
New Roman"; mso-pagination: widow-orphan; mso-fareast-font-family: =
"Times New Roman"
}
PRE {
FONT-SIZE: 10pt; MARGIN: 0in 0in 0pt; COLOR: black; FONT-FAMILY: =
"Courier New"; mso-pagination: widow-orphan; mso-fareast-font-family: =
"Courier New"
}
SPAN.EmailStyle16 {
COLOR: navy; mso-style-type: personal-reply; mso-ansi-font-size: =
10.0pt; mso-ascii-font-family: Arial; mso-hansi-font-family: Arial; =
mso-bidi-font-family: Arial
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext=3D"edit" spidmax=3D"1027"/>
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext=3D"edit">
<o:idmap v:ext=3D"edit" data=3D"1"/>
</o:shapelayout></xml><![endif]-->
<DIV class=3DSection1>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial"><!--[if=20
=
!supportEmptyParas]--> <!--[endif]--><O:P></O:P></SPAN></FO=
NT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">Well, =
I have the=20
same problem, and you do not need a crash to do that. The problem =
is that=20
BPEL prescribe receive-reply as implementing a synchronous WSDL =
operation,=20
when in practice you cannot enforce it. You just need add a wait =
for a=20
week between the receive and the reply, and I’m sure you do =
not want to=20
keep the connection open for that=20
long.<O:P></O:P></SPAN></FONT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial"><!--[if=20
=
!supportEmptyParas]--> <!--[endif]--><O:P></O:P></SPAN></FO=
NT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">I =
opened issue 17=20
(Asynchronous operations) a while back, but have not have time to =
pursue=20
it. IMHO the receive / reply pair does requires an asynchronous =
WSDL=20
binding (one that does not require the connection to remain open). =
In=20
theory, you could define such a binding, but nobody will be able =
to use it=20
because first is not WS-I compliant, and second does not fit most =
WSDL=20
implementation frameworks.<O:P></O:P></SPAN></FONT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial"><!--[if=20
=
!supportEmptyParas]--> <!--[endif]--><O:P></O:P></SPAN></FO=
NT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">It =
may be that=20
WS-Routing provides a solution to this issue by allowing a reverse =
message=20
path for the reply. But, I have not had time to study this=20
alternative.<O:P></O:P></SPAN></FONT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial"><!--[if=20
=
!supportEmptyParas]--> <!--[endif]--><O:P></O:P></SPAN></FO=
NT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial">In =
any case, I’m=20
also interested on see (read) how others are tackling this =
implementation=20
issue….<O:P></O:P></SPAN></FONT></SPAN></P>
<P class=3DMsoNormal><SPAN class=3DEmailStyle16><FONT face=3DArial =
color=3Dnavy=20
size=3D2><SPAN style=3D"FONT-SIZE: 10pt; FONT-FAMILY: =
Arial"><!--[if=20
=
!supportEmptyParas]--> <!--[endif]--><O:P></O:P></SPAN></FO=
NT></SPAN></P>
<P class=3DMsoNormal><!--[if supportFields]><span=20
class=3DEmailStyle16><font size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:
10.0pt;mso-bidi-font-size:12.0pt;font-family:Arial'><span =
style=3D'mso-element:
field-begin'></span><span style=3D"mso-spacerun: =
yes"> </span>AUTOTEXTLIST=20
\s "E-mail Signature" <span =
style=3D'mso-element:field-separator'></span></span></font></span><![endi=
f]--><FONT=20
face=3D"Courier New" size=3D2><SPAN=20
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: 'Courier =
New'">--</SPAN></FONT><FONT=20
face=3D"Courier New"><SPAN=20
style=3D"FONT-FAMILY: 'Courier New'"><O:P></O:P></SPAN></FONT></P>
<P class=3DMsoNormal><FONT face=3D"Courier New" color=3Dblack =
size=3D3><SPAN=20
style=3D"FONT-SIZE: 12pt; FONT-FAMILY: 'Courier =
New'">Regards,</SPAN></FONT><FONT=20
face=3D"Courier New" size=3D2><SPAN=20
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: 'Courier =
New'"><O:P></O:P></SPAN></FONT></P>
<P class=3DMsoNormal><FONT face=3D"Courier New" color=3Dblack =
size=3D2><SPAN=20
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: 'Courier New'">Mike=20
Marin<O:P></O:P></SPAN></FONT></P>
<P class=3DMsoNormal><!--[if supportFields]><span =
class=3DEmailStyle16><font=20
size=3D2 color=3Dnavy face=3DArial><span =
style=3D'font-size:10.0pt;mso-bidi-font-size:
12.0pt;font-family:Arial'><span =
style=3D'mso-element:field-end'></span></span></font></span><![endif]--><=
SPAN=20
class=3DEmailStyle16><FONT face=3DArial color=3Dnavy =
size=3D2><SPAN=20
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Arial"><!--[if=20
=
!supportEmptyParas]--> <!--[endif]--><O:P></O:P></SPAN></FO=
NT></SPAN></P>
<P class=3DMsoNormal style=3D"MARGIN-LEFT: 0.5in"><FONT =
face=3DTahoma=20
color=3Dblack size=3D2><SPAN=20
style=3D"FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">-----Original=20
Message-----<BR><B><SPAN style=3D"FONT-WEIGHT: =
bold">From:</SPAN></B> Ron=20
Ten-Hove [<A class=3Dmoz-txt-link-freetext=20
=
href=3D"mailto:Ronald.Ten-Hove@Sun.COM";>mailto:Ronald.Ten-Hove@Sun.COM</A=
>]<BR><B><SPAN=20
style=3D"FONT-WEIGHT: bold">Sent:</SPAN></B> Tuesday, October 14, =
2003 4:25=20
PM<BR><B><SPAN style=3D"FONT-WEIGHT: bold">To:</SPAN></B> bpel=20
implementation<BR><B><SPAN style=3D"FONT-WEIGHT: =
bold">Subject:</SPAN></B>=20
[wsbpel-implement] Fault tolerance =
considerations</SPAN></FONT></P>
<P class=3DMsoNormal style=3D"MARGIN-LEFT: 0.5in"><FONT =
face=3D"Times New Roman"=20
color=3Dblack size=3D3><SPAN style=3D"FONT-SIZE: 12pt"><!--[if=20
=
!supportEmptyParas]--> <!--[endif]--><O:P></O:P></SPAN></FO=
NT></P>
<P class=3DMsoNormal style=3D"MARGIN-LEFT: 0.5in"><FONT =
face=3D"Times New Roman"=20
color=3Dblack size=3D3><SPAN=20
style=3D"FONT-SIZE: 12pt">Folks,<BR><BR> I was =
recently=20
given an interesting question from one of my development teams, =
and I=20
thought it would be of interest to this group, since it touches on =
universal implementation issues.<BR><BR> The =
question is=20
based on the following scenario: given a process something like=20
this:<O:P></O:P></SPAN></FONT></P><PRE style=3D"MARGIN-BOTTOM: =
0pt; MARGIN-LEFT: 1in; MARGIN-RIGHT: 0.5in"><FONT face=3D"Courier New" =
color=3Dblack size=3D2><SPAN style=3D"FONT-SIZE: =
10pt"><sequence><O:P></O:P></SPAN></FONT></PRE><PRE =
style=3D"MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 1in; MARGIN-RIGHT: =
0.5in"><FONT face=3D"Courier New" color=3Dblack size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt"><SPAN> </SPAN><receive name=3D"rcv" =
... /><O:P></O:P></SPAN></FONT></PRE><PRE style=3D"MARGIN-BOTTOM: =
0pt; MARGIN-LEFT: 1in; MARGIN-RIGHT: 0.5in"><FONT face=3D"Courier New" =
color=3Dblack size=3D2><SPAN style=3D"FONT-SIZE: 10pt"><SPAN> =
</SPAN><assign<SPAN> </SPAN>name=3D"as1" ... =
/><O:P></O:P></SPAN></FONT></PRE><PRE style=3D"MARGIN-BOTTOM: 0pt; =
MARGIN-LEFT: 1in; MARGIN-RIGHT: 0.5in"><FONT face=3D"Courier New" =
color=3Dblack size=3D2><SPAN style=3D"FONT-SIZE: 10pt"><SPAN> =
</SPAN><invoke<SPAN> </SPAN>name=3D"inv" ... =
/><O:P></O:P></SPAN></FONT></PRE><PRE style=3D"MARGIN-BOTTOM: 0pt; =
MARGIN-LEFT: 1in; MARGIN-RIGHT: 0.5in"><FONT face=3D"Courier New" =
color=3Dblack size=3D2><SPAN style=3D"FONT-SIZE: 10pt"><SPAN> =
</SPAN><assign<SPAN> </SPAN>name=3D"as2" ... =
/><O:P></O:P></SPAN></FONT></PRE><PRE style=3D"MARGIN-BOTTOM: 0pt; =
MARGIN-LEFT: 1in; MARGIN-RIGHT: 0.5in"><FONT face=3D"Courier New" =
color=3Dblack size=3D2><SPAN style=3D"FONT-SIZE: 10pt"><SPAN> =
</SPAN><reply<SPAN> </SPAN>name=3D"rep" ... =
/><O:P></O:P></SPAN></FONT></PRE><PRE style=3D"MARGIN-BOTTOM: 0pt; =
MARGIN-LEFT: 1in; MARGIN-RIGHT: 0.5in"><FONT face=3D"Courier New" =
color=3Dblack size=3D2><SPAN style=3D"FONT-SIZE: =
10pt"></sequence><O:P></O:P></SPAN></FONT></PRE><PRE =
style=3D"MARGIN-BOTTOM: 0pt; MARGIN-LEFT: 1in; MARGIN-RIGHT: =
0.5in"><FONT face=3D"Courier New" color=3Dblack size=3D2><SPAN =
style=3D"FONT-SIZE: 10pt"><SPAN> =
</SPAN><O:P></O:P></SPAN></FONT></PRE>
<P class=3DMsoNormal style=3D"MARGIN-LEFT: 0.5in"><FONT =
face=3D"Times New Roman"=20
color=3Dblack size=3D3><SPAN style=3D"FONT-SIZE: 12pt">The =
<receive> and=20
<reply> activities are part of a request-response MEP, bound =
to=20
SOAP, so that the request-response is synchronous (uses the same=20
connection for request and response).<BR><BR> =
Simple=20
enough. But suppose that during execution of an instance of the =
above=20
process, somewhere after the <receive> activity is completed =
but=20
before the <reply> activity is done, the BPEL engine =
suffers a=20
crash. Since we have the full state persistence, recovery is =
simple=20
enough. We can therefore finish creating the reply, but this is =
rather=20
useless, since the client connection is lost. =
<BR><BR> =20
So what is the right thing to do under these circumstances? Should =
the=20
engine, upon recovery in this situation, fault the running =
activity?=20
Should it continue to the reply activity, and presumably fault =
because the=20
connection is closed?<BR><BR> What of the client =
program? It sees that the HTTP connection closed while awaiting a =
response=20
to the request. It might reasonably resend the request (HTTP being =
what it=20
is). If this is the expected behaviour, might it not be =
appropriate for=20
the BPEL engine offering the service our client is using to, upon =
recover,=20
"roll back" or otherwise compensate the completed activities in =
the=20
sequence (not shown in the process above), to the point of the=20
<receive> activity, and restart the=20
receive?<BR><BR> I know that some of these =
complexities=20
are the result of using unreliable messaging, and you get what you =
pay=20
for, right? On the other hand, this illustrates some interesting =
states=20
that a BPEL implementation might have to deal with, which aren't =
discussed=20
in the specification. At the very least, we have some unspecified =
faults=20
to deal with -- presumably implementation specific.=20
<BR><BR> So what are other implementers doing in =
this=20
case? Generating a fault of one sort of another, or performing =
more heroic=20
efforts to recover from the crash? I'm just interested in general=20
approaches, since we don't want to require NDAs here! My =
development team=20
is busy trying to create some recovery mechanisms for the scenario =
above,=20
based on some sort of client/server interaction (client retries =
being the=20
most likely sort). These guys are pretty clever, so I wouldn't =
doubt that=20
they could invent something that, in many cases, actually recover =
from the=20
crash scenario above. <BR><BR> Thoughts? Is =
anyone else=20
concerned about crash recovery, perhaps with different=20
=
scenarios?<BR><BR>-Ron<O:P></O:P></SPAN></FONT></P></DIV></BLOCKQUOTE></B=
LOCKQUOTE></BLOCKQUOTE></BODY></HTML>
------_=_NextPart_002_01C39321.F2323D41--
------_=_NextPart_001_01C39321.F2323D41
Content-Type: image/gif;
name="image002.gif"
Content-Transfer-Encoding: base64
Content-ID: <141303613@15102003-0394>
Content-Description: image002.gif
Content-Location: image002.gif
R0lGODlhkQBRAHcAMSH+GlNvZnR3YXJlOiBNaWNyb3NvZnQgT2ZmaWNlACH5BAEAAAAALAAAAACR
AFEAhIGBgf9QUP8zM/+Zmf/MzP9mZv8AAP///wECAwECAwECAwECAwECAwECAwECAwECAwECAwEC
AwECAwECAwECAwECAwECAwECAwECAwECAwECAwECAwECAwECAwECAwECAwP/CLrc/jBKdw4ZIwhh
ujdbMRBENZ1oqq5su1RDwX103QmBeLh87/8RWMBGLN4CpR1wyWw2LDKj1CgYKZ3Y7OoQnXqL1at2
TH4NvmhpuMzGWmbpOBHZrv8OZ7meuLb7U297gkQFYn+HTwRwg4weSIaIf3iNlDQCJZGIeIuVlJeQ
mWV5naQgBKF1FqWrpqCoTKqsqwKur0CKsqwBtbY8B0O5rAO8vVujwaunxXcEyLmfyz2/zrnD0S6T
1LnE1w8HnNqku90q2eGyyuQn3+e5heonx+2ll/AT4POdA/YRzfnb/B7I+0cqXaQBGxJyY0eQComH
+Gy8y1TCX70JuBqqGTbN/wutVwc8EDOn0QjHiDUuhgppagKXklM4AvNibaVIlyhhgjiJZiKZCkAN
sawX1MQLnVJkohnHoOgOp1CdKrCQQYOOpiIvJHy0oNmGEWA5iRjLoQAJhDQCjLiA1oYAEWfb0uBZ
Q+0FrSDE4JERwC7CSxhiBB7GVsQ+ALjWnoEGYGgGwzNqKhqWYUOJUTjMwjBbKNbOTWEIFEhJOQSJ
mR6UzsWj4RHCKy8vERiyYQjkGzouyNCRp2ZsTCw/w7ihhESgG5dX48EFNLVnA6qRk+zMia7zjHnx
KMkGvIPat41HYVLwujEHlQDyvAvuExdRoKM/WIcOv0OSshXiEz9On6F07P/9odaZfh1wZAI7Fw1l
3FTATMTFYeqJ0dtTHhiU0SlBiTWcc8dRZl8Fi9AConTBcThTGP4ZcB8NnZngnjIM6ZXHR4hdNEMB
DSgCjYIMvFiUhvxx5I8lQGmQkIjZ7PJcgRsy+RKJQB54HowzGJLRPm8okRoFsN3U1QxJNJdWBdhx
NJABLT6k5gXACFmDm9edSGaU9ak4lXcPxCdiARNtKQGPX7aSIZFB5meDkgyddwOHJTpZZgVy8tdB
i0ChpWQeBnVV4Tfj+RmElwpcWFRdc3JIoActxkTmm032B4eIAKJZ1G6t1TbeE/GZpdKN3nTXUqCf
DPpBZ4+S5EGwORmg5JD/HDIbHZJ0NpOEmrXE8pZPeTD1wmwUdiCGqMKi2ipHp5JYrrLABGvJs/PN
1x8MoVkg761PwGHhlLhaA2io+AYlT7uG2mCmDcVyUtEi0cn6JKPpHXGkaaCMIuOk39IYXMX2TbXq
scnJ1+oHyM51jIjHKImaqzRUBHJF3RpB753KUoCWbBh8clZqQra11i+lWUYSDmZu4FYSf8GV0lm2
LpwQ0Rp4rOMnMRQXIggfRAxdrzKEUFMMfHZdiGhe81mC1nfJBYJhg5076Vq68WEYWyEOFtipaOMl
mWls12ynA9IGQaZRjUUl+MJUd5LssYXvIbRzjX1IQd8OZMDNEswixchF/4u5ogiGViig1uS3HG65
R8X9yvdHo33EqR0pjr6Hb06+8IY1U/7iUxtnup6GhanLrcHWk8qQKRuN6p4GjRqLVitcsBUNuB1q
G59UQBBULv0UyFPf1MnXG3G79pp2jz3o6nQkfhE1gc/3+WCQDw/h7DOufq+iu/79/DnWjxR6+FMQ
vfHD6x+u4pc+AVaPe7q7nwETob9/ZG+B/WhgOxQIQS5JUBt0qCAKJAUTCmoQAvBrSAE/uMHcnaMP
JNzCbAiimRT6ogvhQKELsbFCbbRwhsxomize8jIc+gAKF1TDDX3YBDL95XI6eB4R3bAXHXqkCiNc
IvEscJauJaRrbHPfKwISAAA7
------_=_NextPart_001_01C39321.F2323D41--
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]