[OASIS Issue Tracker] (MQTT-256) Message Format indication and message metadata in general.

[OASIS Issues Tracker <workgroup_mailer@lists.oasis-open.org>]

    [ https://issues.oasis-open.org/browse/MQTT-256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=61555#comment-61555 ] 

Ken Borgendale commented on MQTT-256:
-------------------------------------

Comments on WD3

Line 88:  We should state this as servers should retain any metadata they do not understand, but may modify or delete metadata they do understand.  This also implies that the server may know about some metadata fields and not others.  This implies that they need to be able to skip over unknown fields.

Line 89: We have a separate issue concerning message expiration.  While this issue should describe the mechanism to carry the expiration data, we should deal with the semantics of expiration separately.

Line 102: I continue to disagree with the basic concept of a closed metadata.  There are many reasons for clients, servers, or intermediate processors of MQTT to add metadata.  We should define the ability to add user defined metadata.  Other metadata designs include three classes of metadata properties:  Those defined by spec, those registered for common use, and those designated for private use.  

Line 104: the spec should not be in the business of describing the internals of either the client or server.  In this case the statement about the boundary between the client library and the client application is a client internal matter.

Line 132: If you have a template which can be reset on use, you effectively have both of these solutions.  I could for instance use a single templace and do a complete replace of the contents each time.  This encoding would also be useful for handling of a message outside of a connection where I need to attach all metadata to the message.

Line 143: I propose an alternative which is that the metatdata be contained within the current topic name.  If the first byte of the topic name is 0xFF (invalid in UTF-8) then the data is interpreted as a delta to the specified template.  If the first byte of the topic name is 0xFE, then the date is interpreted as a complete replacement of the specified template.  The trailing delimeter is thus not necessary.  If the only metadata is the topic name then the existing format can be used.  This does slightly limit the maximum topic name as all metadata combined cannot exceed 64K bytes, but this seems unlikely to be a real world problem.

Line 153:  A publish is actually 3x and not 03.

Line 159: A client should be able to specifiy the number of templates supported from 0 to 256.  The server can be expected to support 256.

Line 165: With the format of the data for an ID only known if the ID is known, it is not possible to skip over unknown metadata.  I propose that we define two type, string and binary.  A string consists of a UTF-8 string with a leading two byte length.  A binary value consists of a 1 byte length and n bytes of data.  A simple rule could be that even IDs are binary, and odd IDs are strings.
Thus to encode a simple enumeration like payload type, you could use 0x020100 where 02 is the ID, 01 is the binary length, and 00 is the value.     

For MessagePayloadType, there is no reason to add another length field for strings.  The point is to define the payload as a UTF-8 string as opposed to binary.  This is useful for presentation in high level languages.  It would also be very useful to specifically call out JSON and XML as payload formats.  

General:
I would rather drop the whole concept of templates and just encode the metadata in the message.  I think the concern over minimal packet size is overblown in almost all cases.  The fact is that many of these MQTT packets end up being sent as separate TCP packets with 100 bytes of TLS and TCP overhead.   The extra problems involved with the fact that messages are no longer self contained entities creates some very large problems.  Some of this is mitigated by having the template reset which allows the sender to publish a self contained message.  This is especialy important if any sender is doing a resend of the message or has any doubt about the state of the connection.  

> Message Format indication and message metadata in general.
> ----------------------------------------------------------
>
>                 Key: MQTT-256
>                 URL: https://issues.oasis-open.org/browse/MQTT-256
>             Project: OASIS Message Queuing Telemetry Transport (MQTT) TC
>          Issue Type: Bug
>          Components: futures
>            Reporter: Peter Niblett
>            Priority: Critical
>
> MQTT 3.1.1 does not provide an architected way of indicating the format of the payload that is passed in a PUBLISH packet. We need to avoid the risk of spec or data bloat, and I'm not advocating adding a complicated header structure that passes schemas, messageType indicators, MIME type strings or the like, but at the moment there is no standard way for a receiver even to be able to tell whether the payload is binary or string. 
> There are three ways that this can be dealt with today:
> 1. The Receiver already knows. It's implementing some bigger standard or program design which has dictated the encoding format (e.g. someone decides that all messages will be JSON)
> 2. There is an indicator embedded somewhere in the Topic name (The Topic name is  the only header we have in MQTT 3.1.1). There isn't a standard convention for this, so topic space designers have to choose their own way of doing it.
> 3. The Receiver assumes one format and has a go parsing it. If that doesn't look right it tries another



--
This message was sent by Atlassian JIRA
(v6.2.2#6258)