mqtt-broker

How it is built

Architecture overview

An MQTT broker connects a very large number of small, often unreliable clients (sensors, vehicles, phones, gateways) through named topics. A client publishes a small message to a topic such as plant1/line3/temp; the broker delivers it to every client that subscribed to a matching topic filter. Publishers and subscribers never know each other. The broker holds the state that makes this reliable for clients that come and go: a session per client (subscriptions, unacknowledged messages, queued messages for the time it is offline), retained messages (the last value of a topic) and a will message (published if the client vanishes). The idea that makes it work is that the protocol is tiny and stateful by design: a header of two to five bytes, three delivery levels, and a keep-alive timer, so the broker can hold very many idle connections on one machine as long as it spends almost nothing per idle connection. MQTT is the OASIS standard, versions 3.1.1 and 5.0. Mosquitto, EMQX, VerneMQ and HiveMQ are the common open brokers.

Parts:

• Connection layer — accept TCP/TLS/WebSocket, parse packets, keep-alive timers — epoll event loop, Length-prefix framing, Varint (LEB128), TLS 1.3, Mutual TLS, Hierarchical timing wheel, Buffer pools for zero-allocation I/O, x509-certificates (no page yet)
• Router — match a published topic against all subscription filters — topic-matching-trie (no page yet)
• Session store — per-client subscriptions, inflight messages, offline queue — Actor model, MPMC bounded queue, Slab allocator
• Delivery guarantees — QoS 0, 1, 2, redelivery, duplicate suppression — Event deduplication window, Message sequence numbers, Idempotency keys, exactly-once-processing (no page yet)
• Persistence — retained messages, sessions, queued QoS 1/2 messages survive a restart — Write-ahead log, Segmented log, fsync, O_DSYNC and O_DIRECT, CRC32C checksums, State-machine snapshots, Point-in-time recovery
• Cluster — spread clients over nodes and route messages between them — Raft, SWIM gossip membership, Primary-backup replication, Health checking, Leases, Fencing tokens
• Protection — rate limits, load shedding, authentication — Token bucket, Exponential backoff with jitter, Graceful draining, Process supervision
• Testing — a broker is a protocol state machine with many edge cases — Deterministic simulation testing, Coverage-guided fuzzing, hdr-histogram (no page yet)

 devices ── TCP / TLS / WebSocket ──►┌─────────────────────────── broker node ──────────────────────────┐
 (CONNECT, keep-alive, PUBLISH,      │ listener ► packet parser ► auth ► rate limit                      │
  SUBSCRIBE)                         │                │                                                  │
                                     │                ▼                                                  │
                                     │   session (per client): subscriptions, inflight window, queue     │
                                     │         │ PUBLISH                              ▲ deliver          │
                                     │         ▼                                      │                  │
                                     │   topic router (trie of filters) ──matches──► session queues      │
                                     │         │ retain / QoS>0                                           │
                                     │         ▼                                                          │
                                     │   retained store + persistent session log (WAL)                    │
                                     └─────────┬──────────────────────────────────────────────────────────┘
                                               │ forward to nodes with matching subscribers
                                               ▼
                                        other broker nodes  (membership by gossip, state by Raft or CRDT)

How the parts fit together

Connection layer

An MQTT connection is a long-lived byte stream. Each control packet starts with a fixed header: one byte of type and flags, then the remaining length as a variable byte integer of one to four bytes, maximum 268,435,455 (MQTT 5.0 spec). That is Length-prefix framing with a LEB128-style varint length, so the parser knows how much to buffer before decoding and can reject an oversize packet from the header alone. Strings (topic names, client IDs) are length-prefixed with two bytes, which caps them at 65,535 bytes.

The design problem is idle connections. A fleet of 100,000 devices sends one message a minute each; almost every socket is silent almost all the time. An epoll event loop with non-blocking sockets and a worker per core (or Erlang's lightweight process per connection, see the actor discussion below) spends memory, not CPU, per idle client, so the budget is bytes per connection: small receive buffers drawn from pools, fixed-size session objects from a slab allocator, and no thread per socket. The keep-alive is the liveness contract: the client states a period in CONNECT, and if the broker sees no packet for one and a half times that period it closes the connection and, if the disconnect was not clean, publishes the will message (spec). With a million clients, the timers are the second cost, which is what a hierarchical timing wheel solves: insert and cancel in constant time, and a tick touches only the slot that expires now.

TLS 1.3 protects the link; devices on constrained hardware pay the handshake once and keep the connection for days, which is why MQTT brokers care about session resumption and a cheap reconnect. Mutual TLS (client certificates) is the common way to give each device an identity, so the broker maps the certificate to a client ID and an access-control list rather than trusting a password.

For device fleets the client certificate is also the client's identity. Certificate chains, expiry and revocation (x509-certificates, no page yet) decide whether a TLS handshake succeeds under Mutual TLS; the broker then maps a field of the certificate (subject or a subject alternative name) to the client ID and to the topic permissions of that device, so a stolen client ID without the key gets nothing. Rotating a certificate must not drop the session: the new connection takes over the session by client ID.

Router

The router answers: given a topic name, which subscriptions match? Topics are /-separated levels; a filter may use + (exactly one level) and # (this and all deeper levels, only last). Topics starting with $ are reserved for the broker and are not matched by a leading wildcard (spec section 4.7.2). The standard structure is a topic-matching-trie (no page yet): one node per level, children keyed by the level string plus special + and # children, and subscribers stored at the node where the filter ends. A publish walks the trie level by level, following the exact child and the + child, and collecting the # child at each node, so cost is proportional to topic depth and the number of matches, not to the number of subscriptions.

Two refinements matter at scale: lookups are far more frequent than subscribes, so the trie is read-mostly with per-node locks or copy-on-write updates; and wildcard-heavy subscriptions (a monitoring client subscribed to #) turn every publish into one more delivery, which the broker should treat as a single fan-out, not a special path.

Session store

A session is the per-client state, kept after disconnect when the client asks for it (clean session in 3.1.1, Session Expiry Interval in 5.0). It holds the subscriptions, the QoS 1/2 messages sent but not acknowledged, and the queue of messages that arrived while the client was offline. Two ownership models exist:

• Shared structure, locks. One table of sessions with fine-grained locks, as in a classic C broker.
• One owner per session. Each session is an actor: its state is touched only by its own process, other parts send it messages. Erlang brokers (EMQX and VerneMQ are Erlang/OTP systems) take this approach, and a crashed session cannot corrupt another one. The cost is a mailbox and a process per client. The mailbox must be bounded, which is exactly the job of a bounded queue when the broker is written in a language without built-in mailboxes.

The queue needs a policy. A phone that is offline for a day must not make the broker hold unbounded messages: queue limits by count and bytes, a message expiry, and a decision per QoS about what to drop (QoS 0 first, oldest first). Mosquitto exposes max_queued_messages and max_queued_bytes for this (Mosquitto configuration manual).

Delivery guarantees

The three QoS levels are a contract per hop (publisher to broker, broker to subscriber), not end to end:

• QoS 0: at most once; fire and forget; no state.
• QoS 1: at least once; the sender keeps the message until PUBACK arrives and resends with the DUP flag if it does not; the receiver may see a duplicate.
• QoS 2: exactly once between the two parties, through a four-packet handshake (PUBLISH, PUBREC, PUBREL, PUBCOMP) in which the receiver remembers the packet identifier until PUBREL. This is a small instance of the techniques in Event deduplication window and Idempotency keys: the packet identifier is the key and the receiver's stored state is the window. See also the general concept of exactly-once-processing (no page yet).

The delivery level of a message to a subscriber is the minimum of the published QoS and the subscription's granted maximum QoS (spec). The packet identifier is a two-byte number reused once its handshake is complete, so the number of messages in flight per direction is bounded; MQTT 5.0 lets each side state it as Receive Maximum, a flow control window that the sender must respect. Inflight messages are held in order, which is the broker's reason to keep a per-session sequence (Message sequence numbers) next to the identifier.

Persistence

What must survive a restart: retained messages, sessions that outlive a connection, and queued or inflight QoS 1/2 messages. QoS 0 messages are never persisted. There are two cost levels:

• Snapshot: keep everything in memory and write the database to disk on an interval. Mosquitto does this with autosave_interval (Mosquitto configuration manual). Cheap and fast, but a crash loses what arrived since the last save.
• Log: append every state change to a write-ahead log in segments, with a CRC32C per record so a torn tail is detected on recovery, and compact it with periodic snapshots of the state. Durability then costs an fsync per batch, so brokers group commit across clients; QoS 1 acknowledgements are sent after the sync, not before.

Which to choose depends on what the broker promises. If a lost QoS 1 message during a crash is acceptable (telemetry that repeats), the snapshot is enough. If the broker is the system of record for commands (unlock this door), the log with group commit is the cost of a real promise.

The log protects against a crash; Point-in-time recovery protects against a bad change that the log faithfully recorded, such as a misconfigured publisher that overwrote thousands of retained messages. A base snapshot of sessions and retained messages plus the archived log segments rebuilds the broker state as of the instant before it, and the restored state can be branched off to inspect before a cut-over.

Cluster

One node holds a very large number of idle connections; beyond that, or for availability, brokers form a cluster. Clients can connect to any node, so the cluster must know, for each topic, which nodes have a matching subscriber, and forward published messages there. The design questions:

• Membership and liveness: who is in the cluster and who is down. SWIM gossip membership scales to large clusters; for small ones, Health checking with a failure detector is enough.
• Subscription table: either replicated on every node (publish routing is local, subscribing is expensive) or partitioned (subscribing is cheap, publishing costs a lookup). Replicating through Raft gives a consistent table but makes every subscribe a consensus write; eventually consistent replication scales better and tolerates a short window in which a new subscription is not yet known everywhere.
• Session takeover: a client that reconnects to a different node must get its old session. The old node's claim is bounded by a lease and the new node takes over with a higher fencing token, so an isolated old node cannot keep delivering to the same client ID.
• Shared subscriptions (5.0): a $share/group/filter subscription delivers each message to one member of the group, which turns the broker into a work queue for backend consumers. The group state is part of the cluster table.
• Hot standby: for a single-node deployment with a second machine for availability, primary-backup replication of the session log is simpler than a cluster.

Protection

A broker is exposed to the internet and to buggy firmware. A token bucket per client or per IP bounds message and connect rates. Devices that lose their connection reconnect, often all at once after an outage, so clients should use exponential backoff with jitter and the broker should shed connects beyond its handshake capacity rather than fall over. On deploy, graceful draining stops accepting new connections, tells clients to reconnect elsewhere (in 5.0, the DISCONNECT can carry a server reference) and waits for inflight handshakes before exit; Process supervision restarts a crashed node.

Testing

The broker is mostly a state machine over packets and timers, which suits Deterministic simulation testing: run the broker with simulated clients, a simulated clock and a seeded fault schedule, and check invariants (no QoS 2 duplicate, session takeover keeps one owner). The packet parser is the attack surface, so coverage-guided fuzzing of every packet type and property decoder belongs in CI.

hdr-histogram (no page yet) records publish-to-delivery latency per QoS level and the time an inflight QoS 1 or 2 message waits for its acknowledgement, in log-linear buckets with bounded relative error. Load tests and the production metrics read p99 and p999 from the same fixed-size structure, so a slow subscriber or a disk stall at fsync shows up as a tail, not as a shifted mean.

Data and control flow

A QoS 1 publish to three subscribers:

1. The publisher sends PUBLISH (topic, QoS 1, packet ID 7, payload). The parser reads the fixed header and remaining length, then the topic and properties.
2. Authorisation checks that this client may publish to this topic; the rate limiter charges one token.
3. The router walks the trie and returns the matching sessions, locally and on other nodes.
4. If the retain flag is set, the retained store replaces the topic's last value.
5. For each local match the message goes into the session's queue, with the lower of the two QoS values. For nodes with remote matches, one copy is forwarded per node, not per subscriber.
6. The message is appended to the log (for subscribers with persistent sessions at QoS 1/2), then the log is synced in the next batch.
7. Only then the broker sends PUBACK 7 to the publisher.
8. Each subscriber session sends the message when its inflight window has room, and stores it until PUBACK; on reconnect with a persistent session, unacknowledged messages are resent with DUP set. Time goes to: the sync in step 6 (latency of the disk, amortised by batching), the fan-out in step 5 (proportional to matches), and TLS encryption of each delivered copy.

A device reconnects after a network loss:

1. The device sends CONNECT with its client ID, the session flag and a keep-alive; TLS has already authenticated it.
2. The broker finds the existing session; if it lives on another node, a takeover moves it under a new fencing token.
3. If the previous TCP connection is still half-open on the broker, it is closed, which publishes the will unless the will delay (5.0) is set long enough for the client to come back.
4. The broker sends CONNACK with a session-present flag, then resends unacknowledged messages, then the queued ones.

Scaling

In the order each limit bites:

1. Memory per connection. Idle clients cost buffers and timers; smaller buffers, pooled allocation, a timing wheel, and TLS session reuse.
2. Fan-out per publish. One message to a million subscribers is a million writes: batch writes per core, encode the payload once, and send the same buffer to every socket on a worker.
3. Durability. Group commit raises fsync throughput; a separate disk for the log; relax the guarantee for QoS 0/1 telemetry.
4. Router contention. Read-mostly trie, per-core caches of hot topics, wildcard subscriptions treated specially.
5. Cluster traffic. Forward one copy per node, not per subscriber; use shared subscriptions to turn fan-out into load balancing; partition by topic prefix for tenants.
6. Reconnect storms. Backoff with jitter on clients, a connect rate limit and a listen backlog on the broker.

Failure modes

Real implementations

Sources

• OASIS, MQTT Version 5.0, 2019. spec
• OASIS, MQTT Version 3.1.1, 2014. spec
• Mosquitto configuration manual: mosquitto.conf(5) (queue limits, in-flight limits, autosave).
• Broker documentation: EMQX docs, VerneMQ clustering.