commit-log

How it is built

Architecture overview

A distributed commit log (the model behind Apache Kafka and Redpanda) stores a stream of records in named topics. Each topic is split into partitions; each partition is an append-only log in which every record has a number, its offset. Producers append to the end of a partition. Consumers read from any offset onward and remember their own position, so many independent readers replay the same data at different speeds, and an old record stays readable until the retention rule deletes it. The log is the source of truth; queues, caches, search indexes and warehouses are derived from it.

The idea that makes it fast is to do only sequential work: append at the tail, read from a position, never update in place. Appends go to the filesystem's page cache, reads are served from it or by copying file pages straight to the socket, and durability comes from replicating the partition to other brokers, not from fsync on every write. The idea that makes it scale is that ordering is promised only inside one partition, so partitions are independent and can be spread over brokers.

Parts:

• Log storage — one partition as segment files with sparse indexes and checksummed record batches — Segmented log, CRC32C checksums, Length-prefix framing
• I/O path — appends and reads through the kernel's cache, bytes sent without copying — mmap and the page cache, Zero-copy I/O, Group commit
• Network layer — accepts connections and requests, one reactor per thread — epoll event loop
• Compression — per record batch, so bytes are saved on the wire, on disk and in replication — LZ4, Zstandard
• Partitioning — decides which partition a keyed record goes to — Fast hash functions
• Replication — copies each partition to follower brokers and elects a leader — Primary-backup replication, Raft
• Consumers — group members share a topic's partitions and track positions — Consumer offsets, consumer-group-rebalancing (no page yet)
• Retention and compaction — keeps either a time window or the latest record per key — log-compaction (no page yet), Object storage for old segments, Point-in-time recovery
• Delivery guarantees — duplicate-free producers and atomic multi-partition writes — Idempotency keys, exactly-once-processing (no page yet)
• Operations — keeps brokers running, drains them, measures latency, tests, secures traffic — Process supervision, Graceful draining, hdr-histogram (no page yet), Deterministic simulation testing, Coverage-guided fuzzing, Mutual TLS

 producer ──batch (key hash → partition)──►┌──────────── broker 1 (leader of P0) ───────────┐
                                           │ network threads ─► request queue ─► I/O threads │
                                           │            append ▼                    ▲ fetch   │
                                           │   active segment  00000000000000000000.log      │
                                           │   older segments  + .index  + .timeindex        │
                                           └───────┬───────────────────────────────▲─────────┘
                                         replicate │ (followers fetch)      sendfile │ page cache
                                   ┌───────────────▼──────────┐         ┌───────────┴──────────┐
                                   │ broker 2/3: follower P0  │         │ consumer group       │
                                   └──────────────────────────┘         │ member ► P0 offset   │
 metadata quorum (KRaft / Raft) ◄── topics, leaders, ISR changes        └──────────────────────┘
 old segments ──tiering──► object store          committed offsets ──► offsets topic

How the parts fit together

Log storage

One partition on one broker is a directory of segment files (Segmented log). Each file is named with the offset of the first record it contains, the first being 00000000000000000000.log, and a read locates the segment by binary search over an in-memory range and then reads from the file offset (source: Kafka docs, "Log"). Only the newest segment is written; older ones are immutable, so retention is deleting whole files and tiering is copying whole files. A sparse offset index adds one entry every log.index.interval.bytes (4 KiB by default), so finding an offset is a lookup, a small scan and then a sequential read. A segment rolls at log.segment.bytes (1 GiB by default) and data is kept for log.retention.hours (168, seven days, by default; source: Kafka docs, "Broker Configs").

The unit written is a record batch, not a record. Its header carries the base offset, a length, the producer id, epoch and base sequence number, attributes (compression, transactional flag) and a CRC-32C over the rest (source: Kafka docs, "Message Format"). Length-prefix framing lets a reader skip batches without parsing them; CRC32C checksums detect torn or corrupted batches on recovery and on the wire; and the same bytes go producer to broker, to follower and to consumer without being rewritten, which is what lets the broker use zero-copy transfers. The alternative, one record per entry, would multiply headers, checksums and syscalls.

I/O path

Kafka's design stance is "don't fear the filesystem": every write goes to the log immediately, but only into the kernel page cache, and the broker does not keep its own in-process cache, because that would store data twice and add JVM heap pressure (source: Kafka docs, "Design"). That is mmap and the page cache used as the buffer pool. The broker's flush interval defaults to never (log.flush.interval.messages is the maximum long) so the operating system writes dirty pages back, and durability rests on replication instead (source: Kafka docs, "Broker Configs").

Reads use Zero-copy I/O: sendfile moves pages from the cache to the socket in the kernel, replacing four copies and two system calls with one call. The design docs observe that a cluster whose consumers are mostly caught up shows no disk reads, because data is served from cache. The catch is TLS: user-space encryption forces the data through the application, and Kafka does not use sendfile when SSL is on (same source), so encrypted traffic loses this advantage unless the kernel does the TLS (kTLS, covered on the zero-copy page).

Group commit shows up in two places. On the client, the producer batches records for a partition until batch.size is reached or linger.ms elapses (5 ms by default since Kafka 4.0), and compresses the whole batch (source: Kafka docs, "Producer Configs"). On the broker, one append covers every record in the batch, and a produce request carries one batch per partition. The effect is that throughput grows with load while the number of syscalls and replication round trips per record falls.

Network layer

Brokers are I/O-bound servers with many connections, so they use a reactor: acceptor and network threads multiplex sockets with epoll and pass parsed requests to a separate pool of I/O threads, which touch the log (source: Kafka docs, num.network.threads default 3 and num.io.threads default 8). Redpanda takes the other design, thread-per-core: each application thread is pinned to a core, owns a share of the data, and communicates with other threads by message passing rather than locks, using the Seastar library (source: Redpanda docs, "How Redpanda works"). The choice is between the broker sharing structures behind locks and queues (simpler, scales to many cores with tuning) and owning data per core (less contention, more design discipline).

Compression

Compression works on a whole batch, so the producer compresses once and the broker stores and forwards the compressed bytes. The batch attributes support gzip, snappy, LZ4 and Zstandard (source: Kafka docs, "Message Format"). LZ4 costs little CPU and fits low-latency pipelines; zstd gives a better ratio at more CPU and fits cross-datacenter links or long retention. Larger batches compress better, so batching and compression reinforce each other.

Partitioning

A keyed record goes to the partition chosen by a hash of its key; without a key the producer fills a partition up to batch.size before switching to the next (source: Kafka docs, partitioner.class). A fast hash function is enough, but it must be stable across clients and releases, because changing it, or the partition count, remaps keys and breaks per-key ordering for existing data. The partition count is therefore chosen with headroom: it is the upper bound on parallel consumers in a group and the unit of load spread across brokers.

Replication

Each partition has a leader and followers, who replicate by fetching from the leader like a consumer. A write is committed once all replicas in the in-sync set (ISR) have it; a follower that falls behind replica.lag.time.max.ms leaves the ISR; and with acks=all the producer is acknowledged only when the ISR has at least min.insync.replicas members (source: Kafka docs, "Design" and "Broker Configs"). A committed record is not lost as long as one in-sync replica survives. This is Primary-backup replication with a dynamic backup set, as opposed to a majority quorum: with replication factor f+1 it tolerates f failures instead of f out of 2f+1, at the price that the ISR membership must itself be agreed on. If all in-sync replicas are gone, the operator chooses between waiting for one to return and an unclean election of an out-of-sync replica that may lose data (unclean.leader.election.enable, off by default).

That agreement is the job of a consensus quorum, Raft. Kafka's controllers form a metadata quorum (KRaft) that stores topics, leaders and ISR changes, replacing ZooKeeper; a typical deployment uses three or five controllers and needs a majority alive (source: Kafka docs, "KRaft"; KIP-500). Redpanda uses Raft for the data itself: every topic partition is a Raft group with one elected leader and followers, tolerating f failures out of 2f+1 replicas (source: Redpanda docs). Raft per partition gives a simple majority rule and no ISR bookkeeping; the ISR design lets a partition stay writable with fewer replicas than a majority when the operator chooses, and a slow follower costs less because it is simply dropped from the set.

Consumers

A consumer's state is one integer per partition, the offset of the next record, so acknowledgements are cheap, and a consumer can rewind to re-consume after a bug fix (source: Kafka docs, "Design"). Consumer offsets are committed to an internal compacted topic, and the delivery is at-least-once when the commit follows processing.

Consumers in a group split the partitions among themselves, and consumer-group-rebalancing (no page yet) decides who reads what and moves partitions when members come and go. The classic protocol runs the assignment in a client and stops the group at a barrier. Since Kafka 4.0 a new protocol (KIP-848) moves assignment to the broker and makes rebalances incremental, without a global synchronization barrier; it is used when group.protocol=consumer is set (source: Kafka docs, "Consumer Rebalance Protocol").

Retention and compaction

A topic keeps data in one of two ways. Time or size retention deletes the oldest segments. log-compaction (no page yet) keeps the latest record per key: a background cleaner rewrites segments, a record with a null value is a tombstone that deletes the key and is itself removed after a retention period, and cleaning is throttled so it does not starve producers and consumers (source: Kafka docs, "Design"). Compacted topics hold state (current user profiles, table changelogs), and replaying one rebuilds a table.

Long retention on local disks is costly, so recent versions can tier: completed segments move to an Object storage system while the active segment stays on the broker, with separate local and total retention settings (source: Kafka docs, "Tiered Storage", KIP-405). Because the log is immutable and tiered files are whole segments, a topic archive plus its retained tail gives Point-in-time recovery: a consumer or a rebuilt cluster can replay from any retained offset. Redpanda's Tiered Storage offloads segments to object storage the same way and keeps metadata of older segments in the object store (source: Redpanda docs).

Delivery guarantees

A producer retrying after a network error could duplicate a record. The idempotent producer fixes this the way Idempotency keys do in an API: the broker assigns each producer an id and de-duplicates by a sequence number sent with every batch (source: Kafka docs, "Design"). Transactions extend it to atomic writes to several partitions, and the batch attributes mark transactional and control batches. exactly-once-processing (no page yet) combines an idempotent producer with transactional offset commits, so a consume-transform-produce loop commits its output and its input position together. This is exactly-once inside the log; a side effect outside it still needs its own idempotency.

Data and control flow

One produced record to one consumed record.

1. The producer picks a partition by key hash, appends the record to a per-partition batch, and sends the batch when it is full or linger.ms expires; the batch is compressed as a unit.
2. The partition leader's network thread reads the length-prefixed request; an I/O thread verifies the CRC, assigns offsets, and appends the batch to the active segment (a write into the page cache).
3. Followers fetch the new batch from the leader and append it; the leader advances the committed offset once the ISR (or the Raft majority) has it. With acks=all the producer is acknowledged now. The design docs put a committed write on the order of 10 ms.
4. A consumer's fetch asks for "records from offset N"; the broker finds the segment and file position, and sends the bytes with sendfile straight from the cache.
5. The consumer processes the batch and commits offset N+k to the offsets topic.

A broker is restarted for an upgrade.

1. The supervisor sends SIGTERM; the broker drains: it hands over partition leadership, stops accepting new work and finishes in-flight requests.
2. Followers elect new leaders from the ISR; clients refresh metadata and move to them.
3. The restarted broker recovers the tail of its logs (checksums verify the last batches), rejoins as a follower, fetches what it missed and re-enters the ISR; leadership moves back later.

A consumer joins a group.

1. The new member sends a heartbeat to its group coordinator, a broker.
2. The coordinator computes a new assignment (server-side in the new protocol) and gives the members their changes incrementally.
3. Each member fetches the committed offset for its new partitions and resumes from it.

Scaling

In the order the levers are pulled:

1. Producers and consumers. Raise batch size and linger, turn on compression, keep consumers caught up so fetches hit the page cache.
2. Partitions. More partitions give more parallel consumers and spread load; each costs file handles, replication work and recovery time, so add them deliberately.
3. Brokers. Add brokers and move partition replicas to them; the memory left over for the page cache decides how many consumers read from cache.
4. Disks and network. Separate log directories over several disks; replication multiplies inbound bandwidth by the replication factor.
5. Retention. Tier old segments to object storage so local disks hold only the hot tail.
6. Control plane. The metadata quorum and partition counts per cluster bound how large one cluster gets; beyond that, split into clusters and mirror between them.

Failure modes

Real implementations

Related system pages: consumers are often a Stateful stream processor; schemas for records are kept in a Schema registry; a database's changes arrive through a Change data capture pipeline; the log with typed events and replay is an Event store; cluster metadata maps to a Coordination store; the log is also the transport for Metrics and Logging pipelines, and old data can sit in a RAM + SSD tiered cache.

Sources

• Apache Kafka documentation (4.3): Design, Log implementation, Message format, KRaft, Tiered storage, Consumer rebalance protocol, Broker configs, Producer configs.
• KIPs: KIP-405 tiered storage, KIP-500 self-managed metadata quorum, KIP-848 next-generation consumer rebalance protocol.
• Redpanda: How Redpanda works.
• D. Ongaro, J. Ousterhout, In Search of an Understandable Consensus Algorithm, USENIX ATC 2014. paper