Dear,
For my thesis i worked on the MQTT protocol and i would like
to share some suggestions and findings I did. For more info
you can contact me. In the next section you can find a part of
my thesis. Feedback is welcome.
1 Reflection on MQTT
The MQTT protocol is in general easy to understand and use, providing flexibility where needed. It is however far from perfect and is lacking at some points. In this section some important notes are made on the MQTT protocol pointing out areas in which it is lacking, inconsistent or inefficient. By intensively using the MQTT protocol while developing the library these points became clear. This section only contains personal insights and is not based upon work of others. Most of the points discussed require a very thorough understanding of the MQTT protocol and it is advised to refer to the specifications for a broader context.
1.1 Fixed header length field
For every single MQTT message the fixed headers length field is of variable length to reduce overhead. The minimum length is 1 byte and the maximum length is 4 bytes. The most significant bit (0x80) of each byte is used as a flag to indicate if the next byte is a part of the message length or of the payload. The maximum length encoded is 28 bits (268 MB).
While the need for compression is defendable it makes decoding the messages hard due to the resulting relative position of the payload. The length field is however the only place this encoding is used; other places don’t encode the length of a message; they are using fixed width length fields. Even harder to understand is why the variable length field is included in fixed length messages. These messages are often 2 or 4 bytes long of which 1 byte will be the length field. The inconsistent way the length is encoded is a big oversight. The only place where the length can be longer than 24 bit (3 byte) is the publish message. Most messages require no length field.
1.2 Omission of length for publish messages
For every instance where a variable length field needs to be transmitted by MQTT, the variable length data is preceded by a length field. The only exception being the payload of the publish message. For this message the length field of the publish message is used to calculate the payload length. Since this is the only instance where a field can be this big, reducing the length field to 16 bit (65536 bytes) as usual would yield a much more consistent packet structure and remove the need for length encoding. Another consequence of setting the maximum length to 256MB is that the receiver needs to allocate 256MB of RAM or buffer the message on a storage device. This will have significant performance consequences. It is also not future proof, since the maximum size is specified it can be reached. An alternative and time proven solution would be to implement a fragmentation algorithm. This solution is already used by TCP, showcasing the possibility of sending endless streams of data.
1.3 Limited length of will message payload
When connecting to the broker a will message can be specified. This message only differs in 2 ways from a regular message. The first difference is that the message will be published by the broker on behalf of the client when the connection is lost unexpected. The second difference is that the payload is proceeded by a 2-byte length field specifying the payloads length.
Other published messages can have a payload size up to 256MB, the will message payload is limited too only 65kB. This inconsistency could be resolved by making the payload field the last field in the connect message or to change the specification making the maximum payload length always two byte, if desired combined with fragmentation.
1.4 Subscription QoS returned by broker
When subscribing to a topic the clients sets the maximum QoS that can be used for the subscription. The broker will return the granted QoS for that subscription in an acknowledge message. The broker is free to choose any maximum QoS that is equal or lower than the one requested. If the broker would choose any other QoS than the one requested by the client it would only serve to provide suboptimal service to the client.
If the broker does not implement QoS 1 or 2 for publishing or if another client publishes to the subscribed topic using a QoS lower than the requested one by the client the broker can still publish to the client with a lower QoS than the one requested by the client without violating the specifications. The QoS requested by the client only sets the maximum QoS the broker can use and leaves the broker free to choose the QoS used for each message.
If, however the broker would only grand a maximum QoS of 0 or 1 when the client requests 2 and at a later stage a message would be published to the topic with QoS 2 the broker can no longer send this message to the subscribed client with QoS 2 as requested by this client because the broker limited the QoS on the subscription. If the broker would have granted QoS 2 could still use QoS 1 or 0 to publish the message if so desired without violating the specifications.
The usefulness of the returned QoS by the broker is further reduces by the fact that every message published will contain its QoS. There is no need to specify the maximum QoS ahead of time. In order to keep the MQTT protocol as simple as possible the returned QoS should be removed.
1.5 Subscription QoS with wildcards
At the topic above “Subscription QoS”, a clear case is made that the returned QoS for a subscription has no use. When the clients use wildcards this becomes even more problematic. It can lead to hidden changes of the QoS used for other topics. Wildcards allow a client to subscribe to multiple topics at once that match the pattern provided by the client. For each topic subscription the broker will return the maximum QoS granted, for a topic filter with wildcards one return value is send for all matching topics, possibly limiting the maximum QoS of other topics in the process.
Changing the maximum QoS of other topics is a hidden side effect of wildcard usage. This will lead to confusion and hard to track bugs.
1.6 Subscribing with wildcards
The MQTT specification is unclear at some points, leading to confusion an unavoidably to mistakes. The most unclear point is wildcard subscription. At no point in the specification it is stated clearly whether the wildcard subscription also subscribes the clients for future topic that are not yet created. Based on listed requirements the intended working can be deduced. The explanation below should be added to the MQTT specifications.
When a client subscribes for a topic it sends a filter, this filter will be stored by the broker for the client. For every message published, the broker checks if the topic matches the client’s filters. The lowest QoS of all the matching filters is used to send the message to the subscribed client. This point should be better described in the MQTT specifications. The sections of the MQTT specification hinting at the correct implementation of the wildcards is at lines 826-830, 1074-1077, 1374 of [mqtt-v3.1.1] (Gupta, 2014)
1.7 No specifications for control
The biggest feature lacking from the MQTT standard is a backend interface. In stark contrast to COAP where protocol control is specified, it remains unspecified for MQTT. The lack of specification prevents cross library configuration of the broker.
A useful feature for backend control would be to gather statistics, access old messages and check upon the current state of the broker. A common way of gathering status information in MQTT is by using system topics which are proceeded by a dollar sign ($). This is common practice but not standardized. There is also no way for a backend to be notified when a new topic is created. In order to keep track of the whole platform an application needs to subscribe to every topic using wildcards, creating massive overhead.
1.8 Quality of Service
Quality of service is an important feature of MQTT, adding a lot of complexity to the platform. The use of QoS on top of TCP is however questionable. TCP guarantees that bytes will be delivered in order. The only way to lose a message is when the TCP connection is broken. TCP does integrate mechanisms to prevent connection loss, even over saturated networks. In networks where packet loss is common MQTT-SN should be used.
The overhead associated with QoS 2 is very low, only 3 messages of 4 bytes. It is important to point out that a TCP datagram is 64 bytes long, for a common Ethernet MTU of 1500 bytes. Most TCP implementations have smart buffers that will aggregate messages together to limit overhead. In situation where MQTT is used, a stable network connection is required, if not MQTT-SN should be used. The stable connection will result in low connection loss, without much need for MQTT intervention to recover lost messages. Overhead of QoS 2 could dramatically be reduced by only initiating the QoS 2 mechanisms when the TCP connection is lost. In most solutions with reliable network connections QoS 0, 1 and 2 will be interchangeable and behave the same.
A possible solution would be to create a FIFO of 2 times the size of the TCP buffer and numbering the messages. Messages pushed out of the FIFO are destroyed, only when the TCP connection is lost the messages inside the FIFO are retransmitted. Using a 16-bit incrementing message id would insure that duplicates can be detected and discarded. This eliminates the need for QoS and saves memory. The way MQTT currently is implemented the number of messages waiting for confirmation can be infinite requiring lots of memory. In the proposed solution the total length of the messages waiting is the length of the FIFO. A graphical representation of the proposed solution is shown below in Figure 4. MQTT QoS alternative.
Figure 4. MQTT QoS alternative
1.9 UTF-8 string encoding
MQTT is first and foremost a machine to machine communication protocol, the use of human readable string is questionable and results in unnecessary overhead. In most use cases the client will have a unique ID, it would be reasonable to create a topic and subtopic with this unique device ID to be able to store device specific information or configurations.
By enforcing the use of UTF-8 strings MQTT forces the client to convert the client ID number to ASCII. By relaxing the specification this could be solved. Using binary data as topic ID’s MQTT would be more machine friendly. The use of wildcards (#, +) and level separators (/) could still be included. In cases where a human readable format is desired it can be used since UTF-8 or ACHII strings can fit the suggested binary format. For the message itself MQTT already uses binary data, as well as for the password. There are no clear reasons why a distinction was made.