Re: [amqp] HTTP over AMQP proposal

[Alan Conway <aconway@redhat.com>]

So I've been thinking about message mapping options. Here's a sketch of an idea. It involves having 2 different ways to map, ideally we'd get it down to only 1 but I'm not sure we can cover all use cases with one (hope to be proved wrong)

Mapping: 2 modes "natural" and "perfect"
NOTE: both modes map (body, content-type) in the same way.

1. "Natural" - for AMQP/HTTP service integration

headers == application-properties (normalized lowercase)
to == URI
subject == :method
reply-to:/correlation-id: - AMQP meaning

Pro: intuitive mapping for interop between AMQP clients and REST services
- use app-props/custom headers for their usual purpose
- to == URI, subject = method is fairly intuitive
- normal AMQP request/reply semantics (client and server)

Con: does not maintain end-to-end AMQP semantics
- intermediaries may modify/add/remove headers => changes to app-properties
- AMQP-bridge may modify reply-to/correlation-id for cross-protocol correlation and routing
- cannot support signing

2. "Perfect" - for HTTP over AMQP tunnels

message-annotations: { http-map: { ... headers and pseudo-headers ... } }

http-map: keys are descriptors (symbol or int); values are string.
- pseudo-headers: lowercase symbol starting with ':' (':method', ':scheme', ':path') OR pre-defined int value for standard headers
- custom headers are lowercase symbol, don't start with ':'
- any other key type, or unrecognized value, is an error.

Pro: perfect AMQP and HTTP semantics
- No changes to immutable AMQP message, can be signed
- No hijacking/overwriting of AMQP properties
- No confusion between AMQP app-props and HTTP-headers

Con: less intuitive for AMQP developers
- AMQP message props (to:, subject:, reply-to:) are ignored for HTTP routing

- AMQP app-props are not part of the mapping

- Unlikely that existing clients/servers can be swapped between AMQP and HTTP without retooling the other end

RATIONALE for two mappings:

The "natural" mapping allows transparent interop. AMQP clients use AMQP as normal and are unaware that the remote may be HTTP, and vice versa. Allows for flexible development and replacement of mono-protocol clients and services in a mixed HTTP/AMQP.

The "perfect" mapping allows HTTP to be tunneled over an AMQP network with full AMQP semantics including message signing and full use of AMQP properties as required by the AMQP network. It requires both ends of the tunnel to understand this spec.

A tunnel or bridge MAY support both: the presence of the 'http-map' message annotation indicates "perfect" mode, absence indicates "natural mode"

On Tue, Sep 25, 2018 at 11:00 AM Alan Conway <aconway@redhat.com> wrote:

I've been working on something closely related but from a different perspective - bridging between HTTP and AMQP clients/servers in a mixed HTTP/AMQP application.

There's a prototype and some information here: https://github.com/alanconway/envoy-amqp/blob/master/README.md

There's a lot of overlap, I think we can put together a single spec for "HTTP-over-AMQP and HTTP-AMQP bridging". The main addition to what Clemens has laid out is to specify how to use "half-bridges" that go from AMQP to HTTP, or HTTP to AMQP, as well as the complete HTTP-over-AMQP tunnel. This should all be possible using common message mappings, correlation techniques etc.

On Tue, Sep 25, 2018 at 10:45 AM Clemens Vasters <clemensv@microsoft.com> wrote:

Iâve been asking around on LinkedIn and Twitter whether people would find the idea of the HTTP semantics and content model over AMQP attractive, and the response was quite positive.

Â

Asking the question was motivated by my work on AMQP Management which already borrows heavily from the RFC7231 specification. A cleaner approach would be to do a more complete mapping of RFC7231 over AMQP and then use that.

Â

Iâm specifically referring to RFC7231 because that specification describes the semantics and content model of HTTP, which is the most popular foundation for request/response interactions over networks.

Â

The base spec, RFC7230, describes the connection model and wire representation of HTTP. A HTTP-over-AMQP mapping would replace RFC7230 much like HTTP/2 (RFC7540) does â with the differences that HTTP-over-AMQP Â

• is fully bidirectional allowing requests to be issued by either of the connected peers (similar to CoAP; RFC7252),
• request/reply traffic works alongside and multiplexed with any other AMQP traffic,
• request and replies integrate with existing AMQP infrastructures, including routing through queues and pub/sub

Â

Because AMQP is a binary protocol with multiplexing support, an HTTP-over-AMQP mapping ought to provide efficiency gains compared to HTTP 1.1 similar to HTTP/2, even though it should not be considered to be a goal to beat HTTP/2 with respect to byte-on-wire footprint or latency.

Â

The greatest strength of such a combination is that the mapping would optionally reseat HTTP on top of a protocol foundation that is designed for application-guided âLayer 7â routing, including routes via durable entities like queues, and for which the TC is already formalizing new routing capabilities (See summary at http://vasters.com/blog/amqp-the-road-ahead/ ).

Â

The mapping would permit any existing HTTP programming model abstraction to function as-is, with the internal implementation having to be adapted to the wire specification much like those implementations had to adapt to HTTP/2. The new opportunity for frameworks is that an HTTP server request context could allow issuing requests back to the client (and any client could act as server in the conversation) because the underlying protocol is fully symmetric after the initial connection handshake. It would indeed also be possible for a party to initiate a connection, e.g. to come out from behind a NAT, and then only act as server towards its peer.

Â

An HTTP-over-AMQP specification would normatively describe the points in the following outline:

Â

• How is the HTTP message (RFC7230 Sec 3) mapped to an AMQP message
  • Where do elements of the start-line (request-line or status-line) go?
    • method / status-code => msg:properties:subject
    • request-target => msg:properties:to
  • How are headers mapped?
    • End-to-end immutable headers into application-properties (some special headers into properties)
    • Intermediary-contributed/-targeted headers into message-annotations
  • How is the message body represented?
    • Content-Type => msg:properties:content-type
    • Content-Encoding => msg:properties:content-encoding
    • Body => msg:data
  • How does chunked transfer coding work for large messages and streaming?
    • Chunks break across grouped message sequences (msg:properties:group-id, msg:properties:group-sequence). The first message is a group, with group-sequence=0, carries all metadata. Subsequent messages carry minimal metadata and monotonically increasing group-sequence ids with further chunks. The highest order bit of the sequence number acting as FIN bit, marking the last chunk.
• How are further HTTP semantics (expressed in headers) mapped?
  • RFC7231/-32/-33/-34/-35 standard header mappings
    • Content-Type, Content-Encoding, Expires, Date MUST map to respective AMQP standard properties
    • Dictionary compression for remaining standard headers (numeric indexes instead of strings)
  • Data type conversions for well-known header field values (e.g. HTTP Date => timestamp)
  • Application extensions, other RFCs/standards
    • Case-folding of header names to lowercase (AMQP is case-sensitive, HTTP is not)
• How are requests and responses correlated?
  • Reply route
    • Using AMQP Link Pairing
    • Using AMQP Request Annotations
    • Using explicit reply-to destinations (e.g. explicit reply queues) using the new AMQP Addressing model
  • Request correlation
    • req:properties:correlation-id => res:properties:correlation-id (also for chunked replies)
• How do targets, links, and redirects work?
  • HTTP request-target maps to an AMQP node, as link target or anonymous terminus routing target
  • AMQP Addressing (network/container/node)
  • Hybrid HTTP/AMQP Addressing?
• Connection Upgrade from HTTP 1.1; starting AMQP-over-HTTP for http/https URIs
  • This is meant for apps that are written to interact with regular HTTP 1.1 services and which know http/https URIs, and for which their client stack transparently handles upgrades to AMQP-over-HTTP. Â
  • Start w/ GET, OPTIONS or other request, requesting WebSocket upgrade with âamqpâ subprotocol and also supplying a to-be-named HTTP-over-AMQP upgrade request header
    • Allowing other methods is looser than RFC6455, which demands GET. Allowing other methods allows to inline the upgrade path with an existing HTTP API of the service/server.
  • If the server supports HTTP-over-AMQP upgrades and decides it wants to offer the upgrade, it responds with 101 and then initiates the AMQP upgrade per AMQP web socket binding. The 101 response carries a to-be-named HTTP-over-AMQP response header indicating that this isnât just a regular WebSocket upgrade.
  • After the upgrade, it would be nifty for the server to map the initial HTTP 1.1 request to an HTTP-over-AMQP request so that it doesnât have to be resent; the response correlation needs may sabotage that.
• Security model
  • AMQP SASL context plus TLS plus CBS as equivalent to HTTP-over-TLS (RFC7230, 2.7.2)
  • Request-level security carried over from RFC7235

Â

Thoughts?

Â

Â

Â

Clemens Vasters Messaging Platform Architect Microsoft Azure Ã+49 151 44063557 * clemensv@microsoft.com  European Microsoft Innovation Center GmbH |ÂGewÃrzmÃhlstrasse 11Â| 80539 Munich| Germany GeschÃftsfÃhrer/General Managers: Keith Dolliver, Benjamin O. Orndorff Amtsgericht Aachen, HRB 12066

Â

Â