More on message bundling

["Pim van der Eijk" <pvde@sonnenglanz.net>]

The latest Part 2 draft already has a good start on bundling.

Here are some more ideas, building on that.

 

A relatively simple way of controlling bundling is to view bundles as combinations of a single "primary" user message (the driver) and other "secondary" user messages that piggy back onto that message. The "primary" user message determines the P-mode settings applied by the MSH to the entire bundle.  Assumption: in a bundle, the primary user message is the first message unit and the first message unit can be assumed to be the primary user message.

 

Since user messages are submitted separately and usually not at the exact same time, the MSH should temporarily store submitted user messages to allow combining them later into bundles.  The MSH may already compute the eb3:UserMessage XML header including the eb3:PayloadInfo references to message parts. It may also already perform some CPU intensive ebMS3/AS4 functions such as compressing payloads and perhaps already compute secure hashes, as an optimization to speed up security operations in the security module. The MSH cannot yet assemble the ebMS3 SOAP header and does not perform any security operations such as encryption, since that depends on the Pmode of the primary message that a user message is going to be bundled with, nor does it apply any reliability headers for the same reason.

 

Bundling therefore would be controlled controlled by Pmodes.  Suggested new Pmode parameters:

 

Pmode[].bundling.standalone: values "always", "optional", "never"

 

Indicates whether the user message is to be sent as a standalone message without piggy-backed user messages
 ("always"), may be sent separately but may also be bundled with other user messages ("optional") or is never sent as a standalone message ("never"). An ebMS 3 implementation that strictly conforms to part 1 of the ebMS 3.0 Core Specification and does not support any form of bundling will only support the option "always" for this parameter. A parameter value "never" means that the message needs to be bundled with some other message to be sent at all. It cannot act as a primary user message.

 

Pmode[].bundling.bundleswith

 

This parameter is to be specified in case the "standalone" parameter has a value other than "never". Its value is a list of Pmodes.  The semantics of the parameter is that a user message may (if standalone is "optional") or must (if standalone is "never") be bundled with other user messages associated with the Pmodes identified in the list.  The parameter may list other types of Pmodes that describe messages that the user message can be bundled with.  In general these Pmodes SHOULD be destined to the same target MSH (even in a multihop network) and SHOULD satisfy the same or stricter QoS requirements, but the responsibility for making sure this is true is with the Pmode configuration administrator. Typically, the list value of the "bundleswith" parameter for Pmode P1 will also include that same Pmode P1, indicating that more than one user message of this type may be sent.  But if a particular user message is known to be very large or to be sent rarely, the parameter could only list some Pmodes of messages that are known to be small and sent frequently.

 

The MSH should support configuration of the maximum duration of the time it waits for more user messages to be submitted to be added to bundles, the maximum number of user messages it bundles and/or the maximum size of the user message content (including business documents and attachments, after AS4 compression) of the bundle.

 

Pmode[].bundling.maxdelay
Pmode[].bundling.maxmessages
Pmode[].bundling.maxsize

 

Note that the MSH is not required to wait before the maxdelay interval has expired before sending the message. It may send the message before the interval has finished. The maxdelay is a contract with the submitting business application which may have some time-to-perform to be respected.

 

If the "bundleswith" list includes multiple Pmodes, and more than one candidate waiting "primary" user message is available that a user message can be bundled with, the allocation is implementation-dependent.

 

If the value of the "standalone" parameter is "optional", the decision by the MSH to send the message as a standalone message or to bundle it with another user message is implementation-dependent.

 

If the value of the "standalone" is "never", and no primary message is available to bundle the message with within a "maxdelay" duration, the MSH MUST generate an ebms:xxxx BundlingError sender error.

 

Bundling and Pull

 

The description above assumes that submitted user messages are queued temporarily between ebMS processing and between further security and reliability processing.  This means that a submitted user message may not yet be available for pulling, leading to unintuitive results. This could be optimized in advanced implementations as some kind of on-demand bundling, or on-demand MSH completion, triggered by an incoming Pull Request.  Left to implementations.

 

Bundling and Sync

 

The above is really about requests and responses for both Push and Pull binding, but not for response user messages to Two Way / Sync MEP bindings, which must be transmitted on the HTTP back channel without delay.  The synchronous responses to each of the user messages in the bundle should be sent without waiting "maxdelay" for more messages to be bundled with them, to prevent HTTP transport timeouts.

 

Minor comment:  the current bundling chapter writes:

 

"However, when InOrder delivery assurance is required for User message units, bundling SHOULD NOT be in effect for these messages as there is no trace of the submission order on the sending side, and the order of message units under eb:Messaging is not significant."

 

I think we could assume that message units are to be processed in document order.  There is still no guarantee that messages are assigned to the same bundle and/or that different bundles would arrive in-order though, so the warning against bundling is valid.  Perhaps an advanced MSH API could allow bundling or ordering to be controlled. Out of scope for spec.

 

Bundling and multihop

 

Assumption:  intermediaries (such as the "transparent" intermediaries) may not bundle or unbundle messages. The MSH may not know which Pmodes target the same (logical) destination (as this requires knowledge of the routing function). So what can and cannot be bundled cannot be predicted based on Pmode parameters other than the proposed "bundleswith". It seems acceptable to assume that in a multi-hop context any routing function should be able to look at the first user message in a bundle to determine the destination. So the "first unit determines .." principle would apply to both bundling and routing.

 

Pim

---

From: Jacques R. Durand [mailto:JDurand@us.fujitsu.com]
Sent: 08 April 2009 04:18
To: ebxml-msg@lists.oasis-open.org
Subject: [ebxml-msg] follow-up on message bundling

Follow-up on bundling:

 

Here is a short list of issues to be discussed before we start drafting anything on bundling:

 

1. Agree on the packaging of bundles:

- several UserMessages units in  same eb:Messaging?

- what other SignalMessage unit(s)  can go with it?

 

2. What is expected to be common to all messages in the same bundle? Here is a list, by decreasing "obviousness":

- security and reliability.

- the destination. (However: in case of multihop: could a bundle be broken down by the last intermediary, and forwarded to multiple final destinations?) 

- other QoS parameters? (how errors are supposed to be reported, Receipts...)

- the MEP? (could some message units belong to one-way and some to two-way? do we expect two-way responses to be bundled too? what if both parties do not have same bundling capability?)

- the MPC?

Should there be some "profiles" of bundles depending on what is common to their messages?

 

3. Can we pull a bundle?

- should the PullRequests themselves be bundled to indicate intent and ability to get bundles? SHould PullRequest signal be extended? Or could a single, normal PullRequest get a bundled (PMode config)?

 

-jacques

 

---

From: Jacques R. Durand [mailto:JDurand@us.fujitsu.com]
Sent: Wednesday, March 11, 2009 4:29 PM
To: ebxml-msg@lists.oasis-open.org
Subject: [ebxml-msg] an outline of message bundling

Definition:

-------------

Message bundling / chuncking:

(a) ability to group several User Message units in a single ebMS envelope, representing

several logical message units to be delivered separately on reception.

(b) ability to split a large payload over several ebMS envelopes and SOAP messages,

and to deliver by Receiving MSH side as a single unit.

 

Reminder of bundling cases defined in Core Spec:

-----------------------------------------------------------------------

The following is a non-exhaustive list of valid bundling cases:

 

(a) eb:Messaging element with the following children:

an eb:UserMessage element

an eb:SignalMessage element with an eb:PullRequest child

 

(b) eb:Messaging element with the following children:

an eb:UserMessage element

an eb:SignalMessage element with one or more eb:Error children

 

(c) eb:Messaging element with the following children:

an eb:UserMessage element 1840 an eb:SignalMessage element with an eb:PullRequest child

an eb:SignalMessage element (distinct from the previous one) with one or more

eb:Error children

 

(d) eb:Messaging element with the following children:

an eb:SignalMessage element with an eb:PullRequest child

an eb:SignalMessage element (distinct from the previous one) with an eb:Receipt child

 

Bundling of ebMS Messages

-----------------------------------------

 

1- Rationale

 

- performance, simpler QoS processing

 

2- Packaging of bundles

 

- Bundling definition

(several UserMessages units - and SignalMessage units - in same eb:Messaging, NOT several eb:Messaging headers)

- processing model

(idea: semantic equivalent to a sequence of User Messages)

(sending side: before RM and Security, receiver side: unbundle after Security and RM)

- bundling of payloads

(SOAP Body, attachments)

 

3- Controlling bundling

 

- constraints or recommendation on what message units can be bundled together.

(e.g. same MPC? same security credentials? reliability level? MEP and channel binding?..)

- PMode parameters

(bundling based on qty, based on time)

- Bundling of Receipts.

- Bundling of Errors.

 

4- Transfer Considerations

 

- The case of pulled messages

(need to handle this: in a multihop config, a message can be pushed then pulled)

(such bundles must all be assigned to same MPC...)

- Routing bundled messages

(routing info: which UserMessage unit to use?)

 

 

Jacques