[OASIS Issue Tracker] (MQTT-217) Deliver implementation guidance around overlapping subscriptions

[OASIS Issues Tracker <workgroup_mailer@lists.oasis-open.org>]

    [ https://issues.oasis-open.org/browse/MQTT-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=63901#comment-63901 ] 

Peter Niblett commented on MQTT-217:
------------------------------------

This issue is about the following text, contained in 3.3.5 (Publish)

“When Clients make subscriptions with Topic Filters that include wildcards, it is possible for a Client’s subscriptions to overlap so that a published message might match multiple filters. In this case the Server MUST deliver the message to the Client respecting the maximum QoS of all the matching subscriptions [MQTT-3.3.5-1]. In addition, the Server MAY deliver further copies of the message, one for each additional matching subscription and respecting the subscription’s QoS in each case.”

The issue is that the last sentence includes the word MAY. This means that a client that makes overlapping subscriptions could see different behaviour, depending on how the server has been implemented. The reason for having the MAY is that some MQTT implementations have included code to de-duplicate messages sent on overlapping subscriptions, while others have not. 

Note that you can only have overlapping subscriptions if at least one of them uses wildcards.  That's because in 3.8.4 there is text that says
"If a Server receives a SUBSCRIBE Packet containing a Topic Filter that is identical to an existing Subscription’s Topic Filter then it MUST completely replace that existing Subscription with a new Subscription. The Topic Filter in the new Subscription will be identical to that in the previous Subscription, although its maximum QoS value could be different." 

What are the reasons why there might be multiple overlapping subscriptions?  How often do they occur in practice? We have come up with two possible reasons:

i) QOS selection.  Suppose you want to receive messages on a large number of topics, for example X/T1 through X/T999,  and you want to set a maxQoS of 0 for most of them but have a couple of topics (X/T34 and X/T367)  that you want to set with a maxQoS of 2.   You might choose to do this creating a MaxQoS 0 subscription to X/+ followed by MaxQoS 2 subscriptions to X/T34 and X/367.  In this case you'd only want to receive one copy of each message, with the appropriate QoS in each case. 

ii) Multiplexing.  In this case the client is acting on behalf of multiple independent "sub-clients". These could be separate applications, or application objects that register interests in topics. One particular case is where the client is acting as a gateway forwarding messages to and from other devices. The sub-clients or gateway-connected devices might have overlapping subscriptions, and in this case the client would need to forward a copy of a matching message to each of them. The client could achieve this today by creating multiple connections, one for each of its sub-clients and remembering which connection corresponds to which client. It creates the relevant subscriptions on each connection and forwards the messages on to the relevant clients.  However a gateway developer might prefer to multiplex all its sub-clients overt a single connection to the server.

It's not clear that case i) exists in practice -  it could be one that we made up when discussing this problem previously. If it does exist then it would clearly be easier for the clients if the server were to de-duplicate the messages. The server should be able to tell whether it's matched an incoming message against two subscriptions for the same client connection, whereas it is harder for a client.  A client would have to inspect the message payload (since if there are packet IDs they will be different) and it might find it hard to distinguish between a message that matched two subscriptions from one that was genuinely sent twice from the same publisher.  

We certainly have multiplexing gateways as in case ii), and it is possible that they might have overlapping subscriptions - for example one sub-client might only be interested in topic X/T34, whereas another one is interested in them all (e.g. X/+).  This case is rather broken at the moment anyway, regardless of whether the server de-duplicates or not, since the incoming message doesn't indicate which subscription caused it.  If the server de-duplicates, then the gateway client has to run its own mini matching process each time it receives a message from the server to determine which of its sub-clients should receive a copy.  If the server doesn't de-duplicate, the gateway still has the problem of working out who to send the message to. It has to run the matching process itself and also remember which sub-client it has sent to, so that when the next copy arrives it sends it to the next one and so on.  Also if two sub-clients happen to send identical subscriptions (wildcarded or not) then the server effectively de-duplicates, so the gateway has to be aware of this and do its own duplication.. for example suppose that 2 sub-clients subscribe to A/+ and a third one subscribes to A/B.  If a message is published to A/B then the gateway is going to receive 2 copies, but it needs to send it on to all 3 sub-clients. 

Options
1. Leave it as it is, on the grounds that overlapping subscriptions are fairly rare anyway
2. Remove the MAY. It would simplify things for client developers if there's only one behaviour, regardless of the server they have chosen to use. It would help case i) - if that really exists.  It wouldn't help much for case ii). The counter-argument is that it places an extra burden on the server to have to check for possible duplicates, even though they aren't likely to happen in practice, and it also makes things less compatible with other messaging protocols, that the server might also be supporting.
3. Leave the MAY, but introduce a way to relate an incoming message to a client's subscription(s), such as was proposed in MQTT-253. They way to view this is that the server is required to send a logical copy of the message corresponding to each matching subscription. It's free to send each one as a separate PUBLISH packet (identified by a single subscription id), or it can, as an optimisation,  combine some or all of them into a single PUBLISH packet. If it does so, then each PUBLISH packet contains only one copy of the payload, but in the header portion it includes multiple subscription ids, so that they receiver can tell which subscriptions it matches. This would be better for case ii).
4. Leave the MAY but require the server to put some kind of duplicate marker (as suggested by Andy Banks) in the additional messages, if there are any. This would make things easier for the client in case i). 

> Deliver implementation guidance around overlapping subscriptions
> ----------------------------------------------------------------
>
>                 Key: MQTT-217
>                 URL: https://issues.oasis-open.org/browse/MQTT-217
>             Project: OASIS Message Queuing Telemetry Transport (MQTT) TC
>          Issue Type: Improvement
>          Components: futures
>    Affects Versions: 5
>            Reporter: Richard Coppen
>            Assignee: Peter Niblett
>            Priority: Minor
>
> Following on from David Kemper's mqtt-comment on CSPRD02 the TC agreed to open this Jira to deliver implementation guidance around overlapping subscriptions.



--
This message was sent by Atlassian JIRA
(v6.2.2#6258)