Re: [amqp] Size-based link flow control

[Robbie Gemmell <rgemmell@redhat.com>]

I'd similarly be against changing the credit etc meanings here.

I also don't think that its particularly vague, it is clearly message based given the definitions and interactions of 'link-credit', 'available' and 'delivery-count'. E.g in addition to the mentioned detail, 'available' is detailed in the referenced 2.6.7 that it "indicates to the receiver, that the sender could make use of the indicated amount of link-credit", and the actual field definition starts "The number of messages awaiting credit at the link sender endpoint." Separately, also consider the figures in 2.6.7Â+ 2.6.8Â+ 2.6.9 which would essentially be broken if credit weren't message based.

I don't think I would add an additional mechanism either that clients would need to understand to make it useful, to address what seems to mainly be an implementation detail.

Robbie

On Mon, Mar 11, 2019 at 9:00 AM Rob Godfrey <rgodfrey@redhat.com> wrote:

On Sat, 9 Mar 2019 at 00:06, Clemens Vasters <clemensv@microsoft.com> wrote:

The session flow control has frame-size resolution and we also use that to cap memory use in the gateway.

Â

The links end up interacting with the respective link targetâs backend processor/storage. We do have the scenario where a single inbound session carries a bundle of links where each link talks to a different backend. For that, the team wants finer grained flow control metric than a message count.

Â

Â

Von: Alan Conway <aconway@redhat.com>
Gesendet: Freitag, 8. MÃrz 2019 12:46
An: Clemens Vasters <clemensv@microsoft.com>
Cc: amqp@lists.oasis-open.org
Betreff: Re: [amqp] Size-based link flow control

Â

Qpid Dispatch uses the existing byte-based session flow control to bound its memory use. It does effectively block clients that send a lot of bytes - regardless of whether it is many small messages or some very large ones. Why do you need a per-link limit on top of that?

Â

A per-link limit allows senders that create a ton of links to blow up the receiver. Of course a malicious sender could create a ton of sessions, but that's much less likely to happen by accident, and the max number of sessions is easy to limit giving you a guaranteed upper-bound on memory per connection.

Â

Â

Â

On Fri, Mar 8, 2019 at 2:34 PM Clemens Vasters <clemensv@microsoft.com> wrote:

Our engineering team has the strong desire to add a flow control mode where link-credit is counted in bytes rather than in number of messages.

Â

Weâre occasionally getting into the situation where weâre being hammered with a ton of large messages all at once and team is convinced that a byte-oriented credit counter could allow for tighter control of what the client are allowed to push. For session-based flow control, the permitted windows we need are also too big.

Â

The Flow control section 2.6.7. in the core spec is surprisingly vague and doesnât have a hard reference to link-credit being a message counter. The only hard reference seems to be is in 2.7.4. where the link-credit section says that âThe current maximum number of messages that can be handled at the receiver endpoint of the linkâ

When designing AMQP 1.0 originally we deliberately didn't add a literal "bytes" level flow control based on prior experience in earlier protocols - bytes accounting is hard, both in agreeing what is and isn't included in the count (headers, framing overhead, etc) and also in determining behaviour when only partial credit is available. As Alan described, the session-level flow control based on frames was added as a way of performing some form of "byte-like" flow control, frame sizes can be made arbitrarily small if the desire is to closely account for the number of bytes to be sent/received.

If 2.6.7 doesn't explicitly state that link-credit is a message count that would be because it was thought to be obvious that this was the meaning :-), e.g. the definition of link credit in 2.6.7 states:

link-credit

TheÂlink-creditÂvariable defines the current maximum legal amount that theÂdelivery-countÂcan be increased by. This identifies aÂdelivery-limitÂthat can be computed by adding theÂlink-creditÂto theÂdelivery-count.

Only the receiver can independently choose a value for this field. The sender's value MUST always be maintained in such a way as to match theÂdelivery-limitÂidentified by the receiver. This means that the sender's link-credit variable MUST be set according to this formula when flow information is given by the receiver:

link-credit_sndÂ:=Âdelivery-count_rcvÂ+Âlink-credit_rcvÂ-Âdelivery-count_snd.

Which I think makes it pretty clear that that link-credit is to be calculated in terms of the delivery count. Now if you tried really hard to pretend this was ambiguous then you could, I suppose, claim that delivery count only talks of being incrementedÂwhen ever a message is sent rather than explicitly stating "incremented by 1"... but that's a big reach.

I'd be strongly against trying to change the meaning of link credit - this would, I think lead to horrible incompatibilities between implementations from different vendors. If you are absolutely dead set on adding some sort of bytes based flow control, I would suggest making use of the properties field on the flow performative as a way of conveying this additional flow control mechanism (along with some sort of initial connection capability exchange at open so that both sides can declare their ability to use this feature).

-- Rob

Â

Â

So weâre thinking about an extra spec with a layered-on capability that allows for reinterpretation of link-credit and delivery-count as counts of bytes.

Â

Thoughts?

--

_____________________________________________________________________________

Red Hat GmbH,Âwww.de.redhat.com,
Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: ,Charles Cachera, Michael Cunningham, Michael O'Neill, Eric Shander