MQTT Exactly Once is Fine

I was reading the MQTT specification for some light bedtime reading, and I was concerned at the mention of QoS settings, and one specific dread: “Exactly Once”.

Exactly once delivery is a story distributed systems programmers tell each other late at night around campfire. Like teenagers on a date at Make-Out Point, relying on exactly once messaging is the setting for horror stories where machines attempt to talk to one another. That’s because any network message can at some point be dropped, and subsequent messages may arrive, and that these may compromise the state of the system.

The Moral Is: Use Idempotent Messages

At-least-once messaging is usually pretty easy to accomplish. You simply retransmit until you get an acknowledgement. Repeated messages then have no effect. An operation that has the same output or effect, no matter how many times it is repeated is said to be idempotent.

Use this one trick to make your messages idempotent forever!

There’s a little trick in messaging where you can turn non-idempotent messages into idempotent ones by tagging a unique identifier to each message. The other party then remembers which messages it has seen and ignores duplicates. This introduces a problem where the client has to maintain state…but remembering a list of ids can get cumbersome if things go on for long enough.

If you’re using an order-guaranteed network (such as TCP), you can just use an integer for the id. The other party can then maintain the last received id, and ignore messages less than or equal to that id.

MQTT has packet identifiers (ids). The only problem is that MQTT’s ids are 2 bytes for a maximum of 65535 messages in a session. That’s just unrealistic. But then how does MQTT make an at-least-once delivery?

The MQTT two-step

MQTT’s packet identifiers can be randomly chosen, and are stored and released in the session state of the MQTT protocol. The session state must at least last as long as the underlying connection, and can often bridge such sessions.

By default, MQTT messages are “at most once” delivery. You just fire and forget them. But the QoS 1 setting is known as “at least once”. Here’s a demonstration.

1
2
3
4
5
6
7
8
9
10
11
12

+------+-------------------------------+---------+-----------------------------+
| Step | Sender                        | Network | Receiver                    |
|------|-------------------------------|---------|-----------------------------|
|  1   | send PUBLISH QoS=1 Dup=0      |         |                             |
|      |          ID=1234 <message>    |         |                             |
|      | store ID/message locally      |         |                             |
|      |                               |   -->   |                             |
|  2   |                               |         | send message to subscribers |
|      |                               |         | send PUBACK ID=1234         |
|      |                               |   <--   |                             |
|  3   | remove ID/message for ID 1234 |         |                             |
+------+-------------------------------+---------+-----------------------------+

Here we can see the normal flow of a QoS 1 message, but what if we disconnected the client before they received an acknowledgement? In step 1, when we store the message and ID, we add it to a list of “unacknowledged” publish requests. If a publish command is not acknowledged (such as when a connection is broken), the sender MUST resend all the unacknowledged messages again (with Dup=1).

In this way, assuming clients eventually reconnect, all QoS 1 messages are delivered at least once, and will stop retransmitting when the 2 messages are received.

The MQTT box step

But what about “exactly once” kinds of delivery (QoS 2)? That requires a bit more orchestration, but it is not magic. The sender is required to hold on to the message and ID for a bit longer, and the receiver has to keep the ID around as well. Here’s the normal flow of a PUBLISH request under QoS2:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

+------+-------------------------------+---------+------------------------------+
| Step | Sender                        | Network | Receiver                     |
|------|-------------------------------|---------|------------------------------|
|  1   | send PUBLISH QoS=2 Dup=0 \    |         |                              |
|      |          ID=1234 <message>    |         |                              |
|      | store ID/message locally      |         |                              |
|      |                               |   -->   |                              |
|  2   |                               |         | store packet ID              |
|      |                               |         | send message to subscribers  |
|      |                               |         | send PUBREC ID=1234          |
|      |                               |   <--   |                              |
|  3   | remove message for ID 1234    |         |                              |
|      | send PUBREL ID=1234           |         |                              |
|      |                               |   -->   |                              |
|  4   |                               |         | remove packet ID             |
|      |                               |         | send PUBCOMP ID=1234         |
|      |                               |   <--   |                              |
|  5   | remove packet ID              |         |                              |
+------+-------------------------------+---------+------------------------------+

Here, we can see how this is a bit more complicated now than the base case. When the sender stores an ID/message or an ID, they are storing it as an “unacknowledged message” in step 1, and an “unacknowledged release” in step 3. Again, clients MUST resend anything that’s unacknowledged when they reconnect, which means that any duplicated messages can be safely ignored. And at the end of a successful transaction, the session on either party (sender or receiver) does not need to remember which IDs have completed the 4-message box step.

Conclusion: Danger still lurks

So, while MQTT safely implements at most once, at least once, and exactly once delivery guarantees for its messages under the assumption that all clients eventually successfully reconnect, that doesn’t mean that MQTT provides guarantees itself on the order of those messages. Outside the context of a single message, the only order that is guaranteed is that messages from the same publisher will arrive in the same order they are published.

If a program relies on always seeing the cause of action before its effects by a third party, and relies on the MQTT ordering mechanism to guarantee that, most single servers will happily oblige. It is unlikely that a single server forward a request from one client before finishing forwarding another message. But in a bridged or clustered environment, the only thing that has to hold true is that each client’s ordering must be respected. You can see effects before causes in a compliant MQTT system.