patch-delivery

How it is built

Architecture overview

A patch-delivery system gets a new build of a very large, mostly unchanged set of files onto millions of machines while moving only what changed. The publisher side splits every file of a build into chunks, hashes them, compares them with the chunks of the previous build, and uploads only the new ones together with a small signed manifest that lists the chunks of every file. The client side downloads the manifest, works out which chunks it lacks, fetches those from a CDN, builds the new files next to the old ones and swaps them in. The one idea that makes it work is that the unit of transfer is a content-addressed chunk, not a file and not a version: any two builds share most chunks, so an update is the set difference between two manifests. Steam's SteamPipe is the reference (documented in the Steamworks docs, closed source). Open designs of the same shape are casync and zsync; binary-delta tools (bsdiff, Courgette, Zucchini) solve the same problem for a different data shape, executables.

Parts:

• Build splitter — cuts each file into chunks, hashes them and reuses the chunks of the previous build — content-defined-chunking (no page yet), Content-addressed storage, Cryptographic hashes, rolling-hash (no page yet)
• Delta engine — optional second path that diffs old and new bytes (executables, small updates) instead of comparing chunks — binary-delta-diff (no page yet), Suffix array, LZ77, Delta encoding
• Chunk store and compression — compressed chunks (and encrypted ones in Steam) behind a key — Zstandard, Dictionary compression, Object storage, Envelope encryption
• Manifest and trust — per build, the list of files and chunk hashes, signed so a client can tell fresh and genuine from stale or forged — Signed update metadata, Ed25519 signatures, key-rotation (no page yet)
• Distribution — CDN edges, range requests, optionally peers, serving chunks to the client — CDN and edge, HTTP caching semantics, HTTP range requests, TLS 1.3, peer-to-peer-swarming (no page yet), Token bucket
• Client updater — diffs manifests, downloads missing chunks, rebuilds files, commits atomically — Backoff with jitter, fsync and direct I/O, atomic-file-replacement (no page yet)
• Release control — build ids, branches, rollbacks and the state behind them — Primary-backup replication, Point-in-time recovery, Leases, Health checking
• Verification — keeps the patch tools correct against hostile or malformed input — Coverage-guided fuzzing, Deterministic simulation testing
• Operations — keeps the publishing and serving services running and measured — Process supervision, Graceful draining, Mutual TLS, hdr-histogram (no page yet)

 build machine                 publisher backend                      edge / CDN               player machine
┌──────────────┐ chunks+hash ┌───────────────────┐   chunk bytes    ┌─────────────┐ range/GET ┌───────────────────┐
│ content dir  │────────────►│ chunk store       │─────────────────►│ cache (LRU) │──────────►│ updater           │
│ splitter     │ new chunks  │ (object storage)  │                  └─────────────┘           │  1 fetch manifest │
│ (vs. prev    │ only        ├───────────────────┤  signed manifest        ▲                  │  2 diff vs local  │
│  build)      │────────────►│ manifest + build  │──────────────────────── │ ────────────────►│  3 fetch chunks   │
└──────────────┘             │ ids, branches     │   (small, no-cache)     │                  │  4 build new files│
                             └───────────────────┘                    peers (optional)       │  5 verify, swap   │
                                    ▲ "set live"/rollback                                      └───────────────────┘
                                    └── release console

How the parts fit together

Build splitter

The splitter turns a content directory into a sequence of chunks per file, each named by its hash (Content-addressed storage, hash from Cryptographic hashes). The Steamworks docs state the contract plainly: SteamPipe splits each file into chunks of roughly one megabyte, compresses and encrypts each chunk before upload, and when it builds an update it searches for chunks that match the previous build, so ideally only new or modified portions of a file become new chunks (source: Steamworks, "Uploading to Steam"). The docs do not say whether the boundaries are fixed offsets or content-defined; do not assume either. The choice matters: fixed-size cuts break on every insertion (all later chunks shift), while content-defined cuts (content-defined-chunking, no page yet, built on a rolling-hash, no page yet) place boundaries from the bytes themselves so the same data produces the same chunks at any offset. casync takes that second route: it chunks with the buzhash rolling hash, names chunks by SHA-512/256 (or SHA-256) and compresses with zstd (source: casync README). Its extra trick is to remove file boundaries before chunking, so small files are grouped with their neighbours and chunk sizes stay even.

The contract with the rest of the system is the manifest: for each file, an ordered list of (chunk hash, size). Anything that changes the bytes of a chunk without changing the content (timestamps, absolute offsets in a pack file table of contents) creates a new chunk, which is why Steam's guidance to developers is about data layout (keep pack files small, group assets by level or feature, avoid timestamps and original file names, keep table-of-contents offsets local) and not about the transport.

Delta engine

A chunk diff is coarse: change 8 bytes inside a 1 MB chunk and the whole chunk is new. The Steam docs give the failure case: a table of contents full of absolute offsets means one inserted asset changes many small numbers, each lands in a different chunk, and clients can end up downloading over half of a large pack file. When that is the dominant cost, a byte-level delta (binary-delta-diff, no page yet; the alternative is the rsync delta algorithm, no page yet) is the second tool.

• bsdiff finds long matches between old and new with a suffix array and stores the differences as bytewise subtraction, then compresses them. Its author states memory use of max(17n, 9n+m) bytes for generation (n old file size, m new) and n+m for applying, and O((n+m) log n) generation time (source: Colin Percival, bsdiff page). Generation is heavy and run once on the publisher; applying is cheap on the client.
• Courgette (Chrome) and Zucchini (also in the Chromium tree) add knowledge of executables: they disassemble the code far enough to find internal references (jump and call targets), normalise those, diff the result and rebuild. On the Chrome update they quote (190.1 to 190.4), a full update was 10,385,920 bytes, a bsdiff patch 704,512 and a Courgette patch 78,848 (source: Chromium Courgette design page). Their Zucchini notes also give the cost split that decides where such a tool may run: patch generation took roughly 15 to 20 minutes and 1 GB of RAM for Chrome, acceptable once per release on a server, while application runs on the client under tight time, memory and disk limits and must fall back to downloading the full file if patching fails (source: Zucchini README).
• zsync is the third shape: the rsync algorithm run on the client against a block-checksum file published beside the file, fetching only missing blocks with ordinary HTTP range requests, so the server needs nothing but a web server (source: zsync site). It is the model for "no custom server".

The deltas use the same low-level pieces as general compressors (LZ77, Delta encoding). When patches are applied to a store of chunks, the output of a delta can be turned back into chunks and verified against the manifest, so both paths end with the same check: every chunk hash matches.

Chunk store and compression

Chunks sit in Object storage, keyed by hash. Immutability makes caching trivial and makes upload idempotent: a chunk that already exists is skipped. Compress each chunk independently (Zstandard); Steam compresses chunks itself, which is why its docs advise against compressing pack files unless per-asset, because compression that crosses asset boundaries spreads one change over many bytes. A shared dictionary helps when chunks are small and similar (many tiny files); for megabyte chunks the gain is small. Encryption of chunks (Steam keeps all content encrypted and hides non-active versions from customers) is a design choice the docs do not detail; Envelope encryption is the generic pattern if rotating a key must not re-encrypt the store.

Manifest and trust

The manifest is the only object clients must trust: it pins every chunk hash. Steam identifies each depot manifest with a unique 64-bit manifest id and each finished app build with a global BuildID assigned by the master depot server (source: Steamworks, "Uploading to Steam"). A generic design that does not rely on one vendor's channel signs that metadata with threshold roles and expiry, so clients detect rollback (an old manifest replayed), freeze (a stale manifest served forever) and mix-and-match (Signed update metadata). Signatures use Ed25519 (alternative: ECDSA). Signing keys need a rotation plan (key-rotation, no page yet) and the root key stays offline. Chunks themselves need no signatures: if the manifest is genuine and the hash matches, the chunk is genuine, which is what lets untrusted caches and peers serve them.

Distribution

CDN and edge caches serve chunks. Because chunks never change under a given hash, Cache-Control can be far-future immutable (HTTP caching semantics); manifests are the only objects that need short lifetimes or revalidation. Large files fetched piecewise use HTTP range requests; that is exactly how zsync gets only missing blocks from a plain server. Transport is TLS 1.3. For launcher-style systems that need to relieve the origin on a release day, peer-to-peer-swarming (no page yet) adds rarest-first piece exchange; every piece is verified against the manifest hash, so peers need no trust. Steam's docs describe a local content server for testing installs from your own HTTP server, which is the simplest "origin" stand-in.

Token bucket limits the load on the origin and on each client's path: the updater spends tokens per chunk request, so a launcher can run a burst of parallel chunk fetches yet stays at a set average rate, and the CDN or origin applies a bucket per client or per address so a misbehaving launcher cannot drain a release-day budget. The bucket refills with time, not with success, so a client that backs off after errors does not also starve itself of tokens.

Client updater

The updater diffs the new manifest against the local one (or against local file hashes after a "verify files"), downloads the chunks it lacks, writes the new files and commits. Steam builds each new file next to the old one and only then deletes the old files and moves the new ones in, so a 25 GB pack file with a 10-byte change is rebuilt as a new 25 GB file, and the docs name that as the reason to keep pack files to roughly one or two gigabytes (source: Steamworks, "Uploading to Steam"). The commit uses write-new, sync, rename (fsync and direct I/O, atomic-file-replacement, no page yet); a power cut then leaves either the old or the new file. Failed downloads retry with Backoff with jitter, and the update stays resumable because progress is "which chunk hashes are already on disk".

Release control

Builds are immutable; what changes is which build a branch points to. Steam's console lets you set a build live on a beta branch or the default branch and roll back to a previous build in a few clicks, and shows the update size before going live (source: Steamworks docs). The state behind it (build ids, branch pointers, manifests) is small and must not be lost: Primary-backup replication with a Lease for who may change it, Health checking for edges, and Point-in-time recovery to restore the publisher database. Alternatives to a single primary (Raft, Quorum replication) cost more and pay off only when the control plane itself must survive losing a site.

Verification

Patch applicators parse files from the network and feed them to memory-unsafe or complex code (disassemblers, decompressors). Coverage-guided fuzzing of the manifest parser and every delta format is the cheapest defence. Deterministic simulation testing of the updater with injected power cuts and half-written files checks the commit logic.

Operations

Process supervision restarts the build uploader, the manifest publisher and the origin processes when they exit, with a delay and burst limit; a build job that crashes half-way restarts from its uploaded chunks, because chunks are addressed by hash and a re-upload is a no-op. Graceful draining lets an origin or edge node finish the chunk transfers in flight before a deploy, then tell clients to reconnect elsewhere; clients resume, because the updater's progress is the set of chunk hashes already on disk.

Mutual TLS is the identity on the publish path, not on the player path: the build machines, the publisher service and the origin prove a certificate to each other, so only a known pipeline can upload chunks or move a branch pointer. Players still reach the CDN over ordinary TLS 1.3; the signed manifest, not the connection, tells them a build is genuine.

hdr-histogram (no page yet) records chunk-fetch latency and per-download time at the edge and in the updater as percentiles, so a slow tail on one region or one chunk size shows during a release-day surge instead of hiding behind a healthy average.

Data and control flow

Publishing one build:

1. The build machine scans the content directory and splits each file into chunks (about 1 MB in Steam).
2. Each chunk is hashed and looked up against the previous build's chunks; matches are reused. A cache of earlier results speeds repeated builds (Steam keeps .csm and .csd files in the build output folder for this).
3. New chunks are compressed, encrypted and uploaded; unchanged chunks cost no upload.
4. A manifest per depot is written and assigned an id; the backend assigns a build id to the whole build.
5. The release owner sets the build live on a branch. The metadata (manifest ids, expiry) is signed or published; clients see it within minutes (Steam: "within a couple of minutes").

Updating one client:

1. The client learns the branch's current build, fetches the manifest and verifies the metadata.
2. It computes the set difference between wanted and local chunks and sums the download size.
3. It downloads the missing chunks from a CDN edge in parallel; each chunk is verified by hash before use. Time is dominated by the network, then by disk writes.
4. It builds each changed file next to the old one from local plus downloaded chunks.
5. All files done: sync, swap, delete the old ones. The previous state stays usable until this step.

Delta path for a binary: the publisher precomputes a patch per (old version, new version) pair for popular old versions, the client applies it, then verifies the result's hash against the manifest and falls back to chunk download or full download on any mismatch.

Scaling

Pulled in this order:

1. Update size. The first limit is bytes per player. Lever: fix the data layout (stable pack files, no embedded timestamps), then smaller chunks (more reuse, bigger manifests), then a delta path for hot spots such as executables.
2. Origin and edge load. Release day creates one hot set of new chunks. Lever: CDN with immutable cache headers, pre-warming the edges, then peers. Cache hit rate is the number to watch; eviction policy choice (LRU and CLOCK, S3-FIFO, W-TinyLFU) matters when the working set exceeds edge memory.
3. Client disk. Rebuilding large files copies unchanged data (the 25 GB example); this is often slower than the download. Lever: smaller pack files, new files for new content instead of rewriting old ones.
4. Publisher compute. Chunking, compression and patch generation scale with build size. Lever: more cores and separate disks for input and output (Steam recommends a separate disk for build output), then precomputing deltas only for the most common old versions.
5. Manifest size. Small chunks make manifests large for multi-terabyte catalogues; lever: bigger chunks for cold data, tree-of-hashes manifests (merkle-tree, no page yet).

Failure modes

Real implementations

The alternative design: push full files or whole installers and let a CDN absorb the cost. It is simplest and right for small games with rare updates; once a title is tens of gigabytes with weekly patches, bytes per player dominate cost and the chunk-and-manifest design repays its complexity.

Sources

• Valve, Uploading to Steam (SteamPipe: chunking, build steps, pack file guidance): Steamworks docs
• Valve, Depots: Steamworks docs
• systemd, casync: repository and README; L. Poettering, announcement
• zsync: project site
• C. Percival, bsdiff: page
• Chromium, Software Updates: Courgette: design page; Zucchini: README