[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: [OASIS Issue Tracker] (MQTT-234) Shared Subscriptions
[ https://issues.oasis-open.org/browse/MQTT-234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Niblett updated MQTT-234:
-------------------------------
Proposal:
A. Add a new definition to 1.2 Terminology
Shared Subscription
A Shared Subscription comprises a Topic Filter and a maximum QoS. Unlike a regular Subscription, a Shared Subscription can be associated with more than one Session and an Application Message that matches a Shared Subscription is only sent to one of these Sessions' clients. A Session can contain more than one Shared Subscription and can contain Shared Subscriptions alongside regular Shared Subscriptions.
B. Update the descriptions of RETAIN in 3.3.1.3 to say that it applies only to non-shared subscriptions
"When a new non-shared subscription is established, the last retained message, if any, on each matching topic name MUST be sent to the subscriber [MQTT-3.3.1-6]. "
...
"When sending a PUBLISH Packet to a Client the Server MUST set the RETAIN flag to 1 if a message is sent as a result of a new non-shared subscription being made by a Client [MQTT-3.3.1-8]."
....
"Retained messages are useful where publishers send state messages on an irregular basis. A new non-shared subscriber will receive the most recent state."
C. Change the first paragraph of 3.8.3 (Payload for SUBSCRIBE)
Existing text:
"The payload of a SUBSCRIBE Packet contains a list of Topic Filters indicating the Topics to which the Client wants to subscribe. The Topic Filters in a SUBSCRIBE packet payload MUST be UTF-8 encoded strings as defined in Section 1.5.3 [MQTT-3.8.3-1]. A Server SHOULD support Topic filters that contain the wildcard characters defined in Section 4.7.1. If it chooses not to support topic filters that contain wildcard characters it MUST reject any Subscription request whose filter contains them [MQTT-3.8.3-2]. Each filter is followed by a byte called the Requested QoS. This gives the maximum QoS level at which the Server can send Application Messages to the Client. "
Replacement text:
"The payload of a SUBSCRIBE Packet contains a list of Topic Filters indicating the Topics to which the Client wants to subscribe. The Topic Filters in a SUBSCRIBE packet payload MUST be UTF-8 encoded strings as defined in Section 1.5.3 [MQTT-3.8.3-1]. A Server SHOULD support Topic filters that contain the wildcard characters defined in Section 4.7.1. It MAY support Shared Subscription Topic Filters defined in Section 4.9. If it chooses not to support Shared Subscription Topic Filters or Topic Filters that contain wildcard characters it MUST reject any Subscription request whose filter contains them [MQTT-3.8.3-2]. Each Topic Filter is followed by a byte called the Requested QoS. This gives the maximum QoS level at which the Server can send Application Messages to the Client. "
D. Change the behaviour described in 3.8.4 (SUBSCRIBE response) :
Existing text:
"If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3].
Where the Topic Filter is not identical to any existing Subscription’s filter, a new Subscription is created and all matching retained messages are sent."
Replacement text:
"If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to a Non-shared Subscription’s Topic Filter for the current Session then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different. Any existing retained messages matching the Topic Filter MUST be re-sent, but the flow of publications MUST NOT be interrupted [MQTT-3.8.4-3].
Where the Topic Filter is not identical to any Non-shared Subscription’s Topic Filter for the current Session, a new Non-shared Subscription is created and all matching retained messages are sent.
Where the Topic Filter is identical to the Topic Filter for a Shared Subscription that already exists on the Server, the Session is added as a subscriber to that Shared Subscription. No retained messages are sent.
Where the Topic Filter is not identical to any existing Shared Subscription’s Topic Filter, a new Shared Subscription is created. The Session is added as a subscriber to that Shared Subscription. No retained messages are sent.
See section 4.9 for more details on Shared Subscriptions. "
E. Add new section 4.9 called Subscriptions
MQTT provides two kinds of Subscription, Shared and Non-Shared.
4.9.1 Non-Shared Subscriptions
A non-shared MQTT subscription is associated only with the MQTT Session that created it. Each Subscription includes a Topic Filter, indicating the topic(s) for which messages are to be delivered on that Session, and a maximum QoS level. The Server is responsible for collecting messages that match the filter and transmitting them on the Session's MQTT connection if and when that connection is active.
A Session cannot have more than one subscription with the same Topic Filter, so the Topic Filter can be used as a key to identify the subscription within that Session.
If there are multiple clients, each with its own non-shared subscription to the same Topic, each client gets its own copy of the Application Messages that are published on that Topic. This means that non-shared subscriptions cannot be used to load-balance Application Messages across multiple consuming clients as in such cases you would want each message to be delivered to exactly one client.
4.9.2 Shared Subscriptions
A Shared Subscription can be associated with multiple subscribing MQTT Sessions. Like a non-shared subscription it has a Topic Filter and a maximum QoS, however a publication that matches its Topic Filter is only sent to one of its subscribing Sessions. This makes it suitable for the load-balancing case.
A Shared Subscription is identified using a special style of Topic Filter. The format of this filter is:
$share/{ShareName}/{filter}
- $share is a literal string that marks the Topic Filter as being a Shared Subscription Topic Filter.
- {ShareName} is a character string that does not include "/", "+" or "#"
- {filter} The remainder of the string has the same syntax and semantics as a Topic Filter in a non-shared subscription - see section 4.7
A Shared Subscription's Topic Filter MUST start with $share and MUST contain a ShareName that is at least one character long.
Non-normative comment
Shared Subscriptions are defined at the level of the MQTT server, rather than at the level of an individual Session. A ShareName is included in the Shared Subscription's Topic Filter so that there can be more than one Shared Subscription on a server that has the same {filter} component. An application could choose to use the ShareName to represent the group of subscribing Sessions that are sharing the subscription, but it is not obliged to do this.
Examples:
1. Shared subscriptions "$share/consumer1/sport/tennis/+" and "$share/consumer2/sport/tennis/+" are distinct shared subscriptions and so can be associated with different groups of Sessions. Both of them match the same topics as a non-shared subscription to sport/tennis/+ . If a message were to be published that matches sport/tennis/+ then a copy would be sent to exactly one of the Sessions subscribed to $share/consumer1/sport/tennis/+ , a separate copy of the message would be sent to exactly one of the Sessions subscribed to $share/consumer2/sport/tennis/+ and further copies would be sent to any clients with non-shared subscriptions to sport/tennis/+
2. Shared subscription "$share/consumer1//finance" matches the same topics as a non-shared subscription to /finance.
Note that "$share/consumer1//finance" and "$share/consumer1/sport/tennis/+" are distinct shared subscriptions, even though they have the same ShareName. While they might be related in some way, no specific relationship between them is implied them having the same ShareName.
A Shared Subscription is created by using a Shared Subscription Topic Filter in a SUBSCRIBE request. So long as only one Session subscribes to a particular Shared Subscription, the shared subscription behaves like a non-shared subscription, except that
i) The $share and {ShareName}portions of the Topic Filter are not taken into account when matching against publications.
ii) No Retained Messages are sent to the Session when it first subscribes. It will be sent other matching messages as they are published.
Once a Shared Subscription is in existence it is possible for a different Session to submit a SUBSCRIBE request with the same Shared Subscription Topic Filter - this different Session could either come from a different client or from a different connection from the same client. If the server authorizes the SUBSCRIBE request, the subscribing Session gets associated with the Shared Subscription as an additional subscriber. Retained messages are not sent to this new subscriber. Each subsequent Application Message that matches the Shared Subscription is now sent to one or other of the two Sessions that share the subscription - but not both.
Further Sessions can also join the Shared Subscription by sending a SUBSCRIBE request with the same Shared Subscription Topic Filter. These new subscribers don't receive any Retained Messages, however they can expect to receive Application Messages. Each subsequent Application Message that matches the Shared Subscription is sent to one of the larger pool of Sessions that has subscribed to it.
Sessions can explicitly detach themselves from a Shared Subscription by sending an UNSUBSCRIBE packet that contains the full Shared Subscription Topic Filter. Sessions also get detached from their Shared Subscriptions when they terminate.
A Shared Subscription lasts for as long as it is associated with at least one Session (i.e. a Session that has issued a successful SUBSCRIBE request to its Topic Filter and that has not completed a corresponding UNSUBSCRIBE). A Shared Subscription survives when the Session that originally created it leaves, unless there are no other Sessions left when this happens. A Shared Subscription ends, and any undelivered messages associated with it are deleted, when there are no longer any Sessions associated with it.
Further notes on Shared Subscriptions
1. If there's more than one Session subscribed to the Shared Subscription, the server implementation is free to choose, on a message by message basis, which Session to use and what criteria it uses to make this selection.
2. Different subscribing clients are permitted to specify different Requested QoS levels in their SUBSCRIBE request packets. The server is permitted to grant different QoS levels to these subscribers. If this happens, the server MUST respect the granted QoS for the subscribing session when sending an Application Message to that session.
3. If a subscribing client's Connection breaks while the Server is in the process of sending it a QoS 2 Application Message, the Server MUST handle the delivery of the message as described in section 4.3.3, completing delivery to that client when it reconnects. If that client's Session terminates before the client reconnects, the Server MAY start sending it to another subscribed client.
4. If the server is in the process of sending a QoS 1 message to its chosen subscribing client and the connection to that client breaks before the Server has received an acknowledgement from the client, the Server MAY attempt to send the Application Message to a different client that's subscribed to the Shared Subscription.
5. If a client responds with a negative acknowledgement to a PUBLISH packet from the Server, the Server MUST discard the Application Message and not attempt to send it to any other Subscriber.
6. A client is permitted to submit a second SUBSCRIBE request to a Shared Subscription on a Session that's already subscribed to that Shared Subscription - it might for example do this in order to change the Requested QoS for its subscription. This does not increase the number of times that that Session is associated with the Shared Subscription, so the Session will leave the Shared Subscription on its first UNSUBSCRIBE.
7. Each Shared Subscription is independent from any other. It is possible to have two Shared Subscriptions with overlapping filters. In such cases a message that matches both Shared Subscriptions will be processed separately by both of them. If a client has a Shared Subscription and a Non-Shared subscription and a message matches both of them, the client will receive a copy of the message by virtue of it having the Non-Shared Subscription. A second copy of the message will be delivered to one of the subscribers to the Shared Subscription, and this could result in a second copy being sent to this client.
was:
A. Add a new definition to 1.2 Terminology
Shared Subscription
A Shared Subscription comprises a Topic Filter and a maximum QoS. Unlike a regular Subscription, a Shared Subscription can be associated with more than one Session and an Application Message that matches a Shared Subscription is only sent to one of these Sessions' clients. A Session can contain more than one Shared Subscription and can contain Shared Subscriptions alongside regular Shared Subscriptions.
B. Change the sentences in 3.8.3 that talk about RequestedQoS
Existing text:
"Each filter is followed by a byte called the Requested QoS. This gives the maximum QoS level at which the Server can send Application Messages to the Client.
The payload of a SUBSCRIBE packet MUST contain at least one Topic Filter / QoS pair. A SUBSCRIBE packet with no payload is a protocol violation. See section xxx for information about handling errors.
The requested maximum QoS field is encoded in the byte following each UTF-8 encoded topic name, and these Topic Filter / QoS pairs are packed contiguously. "
Replacement text:
"Each filter is followed by a Flags Byte. This indicates whether the filter is part of a shared or an unshared subscription and gives the maximum QoS level at which the Server can send Application Messages to the Client for this subscription.
The payload of a SUBSCRIBE packet MUST contain at least one Topic Filter / Flags Byte pair. A SUBSCRIBE packet with no payload is a protocol violation. See section xxx for information about handling errors.
The Topic Filter / Flags Byte pairs are packed contiguously. "
C. Change the SUBSCRIBE Packet payload format figure to change the name of byte N+1 to be Flags Byte and show its Bit 7 to be a flag called Shared. Replace the text that follows the figure:
Existing Text
"The upper 6 bits of the Requested QoS byte are not used in the current version of the protocol. They are reserved for future use. The Server MUST treat a SUBSCRIBE packet as malformed and close the Network Connection if any of Reserved bits in the payload are non-zero, or QoS is not 0,1 or 2. "
Replacement Text
"Bit 7 of the Flags Byte indicates that this Topic Filter is part of a shared subscription. See section 4.9 for details about shared subscriptions.
Bits 3 through 6 of the Flags Byte are not used in the current version of the protocol. They are reserved for future use. The Server MUST treat a SUBSCRIBE packet as malformed and close the Network Connection if any of these Reserved bits in the payload are non-zero, or if QoS is not 0,1 or 2. "
D. Add the following to 3.10.4 (Response section in UNSUBSCRIBE)
"If the Topic Filter supplied in the UNSUBSCRIBE packet corresponds both to a Shared Subscription and to a regular subscription the Server MUST unsubscribe the Shared Subscription, leaving the regular subscription unchanged.
Non-normative comment
This situation could only occur if the regular subscription has a Topic Filter that starts with the Shared Subscription's name prefix followed by the ":" character. This eventuality can be avoided by avoiding the use of ":" in Topic names, or at least not using it at the end of the first topic level.
E. Add new section 4.9 called Shared Subscriptions
A regular (non-shared) MQTT subscription is associated only with the MQTT Session that created it. If there are multiple clients, each with its own non-shared subscription to the same Topic, each client gets its own copy of the Application Messages that are published on that Topic. This means that non-shared subscriptions cannot be used to load-balance Application Messages across multiple consuming clients as in such cases you would want each message to be delivered to exactly one client. In contrast, a Shared Subscription can be associated with multiple consuming clients and a publication that matches its Topic Filter is only sent to one of these clients. This makes it suitable for the load-balancing case.
A client creates a Shared Subscription by sending a SUBSCRIBE packet with the Shared flag set to true (see section 3.8.3). The Topic Filter in a Shared Subscription differs from the Topic Filter used in a non-shared subscription in that it contains a prefix. This prefix is separated from the remainder of the Topic Filter by a ":" character followed by a topic level separator "/" character.
The remainder of the Topic Filter (i.e. the string following the "/" character) has the same syntax and semantics as a Topic Filter in a non-shared subscription.
The prefix part of the Topic Filter, up to and including the "/" character, is ignored when matching against Topic Names. It forms part of the name of the shared subscription and its purpose is to let there be more than one shared subscription matching a given set of topics - see example 1 below.
Examples:
1. Shared subscriptions "consumer1:/sport/tennis/+" and "consumer2:/sport/tennis/+" are distinct shared subscriptions and so can be associated with different groups of Sessions. Both of them match the same topics as a non-shared subscription to sport/tennis/+ .
If a message were to be published that matches sport/tennis/+ then a copy would be sent to exactly one of the Sessions subscribed to consumer1:/sport/tennis/+ and a separate copy of the message would be sent to one of the Sessions subscribed to consumer2:/sport/tennis/+
2. Shared subscription "consumer1://finance" matches the same topics as a non-shared subscription to /finance.
Note that "consumer1://finance" and "consumer1:/sport/tennis/+" are distinct shared subscriptions. No relationship between them is implied by them having the same name prefix.
So long as only one Session subscribes to a particular Shared Subscription, the Shared Subscription behaves like a non-shared subscription, except that its name prefix is not taken into account when matching against publications. The client receives any Retained Messages that match the Topic Filter when it first subscribes. It will be sent other matching messages as they are published. The Shared Subscription lasts until the Session ends or until the client unsubscribes from it, whichever happens first.
Once a Shared Subscription is in existence it's possible for a different Session to submit a SUBSCRIBE request with the same Topic Filter, including the same prefix - this different session could either come from a different client or from a different connection from the same client. If the server authorizes the SUBSCRIBE request, the subscribing session gets associated with the Shared Subscription. Retained messages are not sent to this new subscriber. Each subsequent Application Message that matches the Shared Subscription is now sent to one or other of the two Sessions that share the subscription - but not both.
Further Sessions can also join the Shared Subscription by sending a SUBSCRIBE request with the same Topic Filter, including the same prefix. These new joiners don't receive any Retained Messages, however they can expect to receive Application Messages. Each subsequent Application Message that matches the Shared Subscription is sent to one of the larger pool of Sessions that has subscribed to it.
Sessions can explicitly detach themselves from a Shared Subscription by sending an UNSUBSCRIBE packet that contains the full shared Topic Filter, including the prefix. Sessions also get detached from their Shared Subscriptions if they terminate.
A Shared Subscription lasts for as long as it is associated with at least one Session (i.e. a Session that has issued a successful SUBSCRIBE request to its Topic Filter and that has not completed a corresponding UNSUBSCRIBE. A Shared Subscription survives when the Session that originally created it leaves, unless there are no other Sessions left when this happens. A Shared Subscription dies, and any undelivered messages associated with it are deleted, when there are no longer any Sessions associated with it.
Further notes on Shared Subscriptions
1. If there's more than one Session associated with the Shared Subscription, the server implementation is free to choose, on a message by message basis, which Session to use and what criteria it uses to make this selection.
2. Different subscribing clients are permitted to specify different Requested QoS levels in their SUBSCRIBE request packets. The server is permitted to grant different QoS levels to these subscribers. If this happens, the server MUST respect the granted QoS for the subscribing session when sending an Application Message to that session.
3. If a subscribing client is using cleanSession = false and its Connection breaks while the Server is in the process of sending it a QoS 2 message, the Server MUST handle the delivery of the message as described in section 4.3.3, completing delivery to that client when it reconnects.
4. If the server is in the process of sending a QoS 1 message to its chosen subscribing client and the connection to that client breaks before the Server has received an acknowledgement from the client, the Server MAY attempt to send the Application Message to a different client that's subscribed to the Shared Subscription.
5. If a client responds with a negative acknowledgement to a message from the Server, the Server MUST discard that message and not attempt to send it to any other Subscriber.
6. A client is permitted to submit a second SUBSCRIBE request to a Shared Subscription on a Session that's already associated with that Shared Subscription - it might for example do this in order to change the Requested QoS for its subscription. This does not increase the number of times that that Session is associated with the Shared Subscription, so the Session will leave the Shared Subscription on its first UNSUBSCRIBE.
7. Each shared subscription is independent from any other. It is possible to have two shared subscriptions with overlapping filters. In such cases a message that matches both shared subscriptions will be processed separately by both of them.
8. Retained messages are sent to the Session that creates the Shared Subscription by being its first subscriber. They are not sent at any other stage during the lifetime of the Shared Subscription, including on subsequent SUBSCRIBE requests from that same Session.
updated proposal following f2f day 2
> Shared Subscriptions
> --------------------
>
> Key: MQTT-234
> URL: https://issues.oasis-open.org/browse/MQTT-234
> Project: OASIS Message Queuing Telemetry Transport (MQTT) TC
> Issue Type: New Feature
> Components: core
> Affects Versions: 5
> Reporter: Andrew Banks
> Assignee: Peter Niblett
> Priority: Critical
>
> Shared Subscriptions
> --------------------
> Shared subscriptions allow a set of clients to share the consumption of messages
> produced in response to a subscription. The set of subscribers is created by the use
> of a common share name. Shared subscriptions provide a facility similar to point to
> point messaging in that the processing of messages is not limited to the availability
> or capacity of a single consumer, the share name is analogous to the queue name in
> that it names the list of messages to be processed.
> I suggest that a shared subscription be defined by the topic filter syntax:
> $share:shareName:topicFilter
> - This specialised topic filter takes the place of the normal topic filter in the
> subscribe packet.
> - The "$share:" prefix uses a reserved $ name and will therefore not cause a conflict
> with other topic filters.
> - shareName is a string which labels the shared subscription and must not contain any
> ":" characters.
> - TopicFilter follows the existing topic filter rules.
> - Subscribing clients will be a member of the share only if both the shareName and
> topicFilter are identical. So, a different shareName with the same topicFiler
> will create a different share. Similarly a shareName followed by a different topic
> filter will create separate share.
> - Servers may decline to support shared subscriptions by not accepting subscribe
> packets containing topic filters starting with the "$share:" prefix. However, if
> they do accept the "$share:" prefix they must follow this definition.
> - The subscribing clients may make both non durable and durable subscriptions by
> setting clean session true or false. The shared subscription ceases to exist if
> the number of clients subscribed to it reaches zero. If the shared subscription
> ceases to exist the messages queued for processing are deleted by the server as
> with an non shared subscription.
> - A good server implementation will avoid allocating messages to subscribers with
> durable subscriptions while the client is disconnected. Instead the messages will be
> allocated when a suitable client connects.
> - A Qos=1 message should be reallocated to another client sharing the same subscription
> if the network connection to the client is lost before the PUBACK is received.
> Qos=0 and Qos=2 messages must not be reallocated.
--
This message was sent by Atlassian JIRA
(v6.2.2#6258)
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]