file-sync

How it is built

Architecture overview

A file-sync system makes a folder on one device appear on others, tolerates devices that are offline for weeks, and never loses a user's data. It is two systems glued together. The content side treats a file as a list of blocks identified by their hash, so only blocks the other side does not have are ever sent, and identical blocks are stored once. The metadata side is a small, strongly ordered database of the folder tree (names, IDs, block lists, versions) that every client reads incrementally with a cursor. Each client runs a loop: notice a local change, break the file into blocks, upload the missing blocks, commit the new block list to the metadata, and, in the other direction, hear that the metadata moved, download the missing blocks and apply the change with a crash-safe write. The one idea that makes it work is the separation of bulky, immutable, content-addressed data from small, ordered, mutable metadata: blocks can be cached, deduplicated and stored anywhere, while correctness (who wins, in what order) lives only in the metadata. Dropbox is the closed example, and its engineers have described the design problems in public; the open examples are Syncthing (peer to peer), Seafile and Nextcloud (server based).

Parts:

• Change detection — notice what changed on disk — File system watching, Cryptographic hashes (SHA-2, BLAKE3)
• Chunking and delta — cut files into blocks and find what is new — content-defined-chunking (no page yet), rsync-delta-algorithm (no page yet), merkle-tree (no page yet)
• Block store — hold the blocks, compressed and encrypted — Object storage, Zstandard (zstd), Envelope encryption
• Transfer — move blocks over bad networks — Resumable uploads, Exponential backoff with jitter
• Namespace and journal — the ordered metadata of the folder tree — Write-ahead log, B+ tree, Incremental sync cursors
• Conflict handling — decide whether two edits are ordered or concurrent — Vector clocks
• Notification — tell a client the journal moved — Server-sent events
• Local apply — write the downloaded file without ever exposing a half file — fsync, O_DSYNC and O_DIRECT

 device A                                              server (or peers)                     device B
 ┌────────────────────┐                      ┌────────────────────────────────┐       ┌────────────────────┐
 │ watcher + rescan   │                      │ namespace service              │       │ watcher + rescan   │
 │   │ changed paths  │                      │  journal (WAL) + B+ tree index │       │                    │
 │   ▼                │  1 upload missing    │  file id → revision → blocks   │  4    │                    │
 │ chunker + hasher ──┼──── blocks ─────────►│ block store (object storage)   │ notify│                    │
 │   │ block list     │  2 commit(parent,    │   hash → compressed, encrypted │──SSE─►│ cursor pull ◄──┐   │
 │   ▼                │     blocklist) ─────►│                                │       │   │            │   │
 │ local state DB     │◄──── 3 ack, rev ─────│                                │◄──────┼── fetch blocks │   │
 └────────────────────┘                      └────────────────────────────────┘       │   ▼            │   │
                                                                                      │ temp file → fsync → rename
                                                                                      └────────────────────┘

How the parts fit together

Change detection

The client has to find out that a file changed without rereading the disk. File system watching uses the OS change notifications, but treats them as hints: events are debounced (an editor save is several events), and whenever the OS reports an overflow, or the client was not running, the client does a full rescan and diffs it against its local state database. The contract with the next part is a set of paths that may have changed; the chunker decides whether the content really did. The local database stores, per file, the size, modification time and block list from the last sync, so an unchanged file costs one stat, not a read. A file that changes while it is being read is detected by comparing size and mtime before and after, and retried.

Chunking and delta

The unit of transfer is the block. Two designs exist and both are in production:

• Fixed-size blocks. The Dropbox API documents its content_hash as: split the file into 4 MB blocks, SHA-256 each block, concatenate the digests and SHA-256 the result (source: Dropbox API docs). That is a Merkle tree of depth one (merkle-tree, no page yet), so two sides can compare whole files by comparing one hash, and find the differing block by comparing block hashes. Syncthing's protocol also uses fixed blocks, 128 KiB to 16 MiB, constant within a file and chosen per file (source: Syncthing BEP spec).
• Content-defined chunking. content-defined-chunking (no page yet) cuts where a rolling hash of a small window (restic uses a 64-byte window with Rabin fingerprints; restic design doc) meets a condition, so an insertion near the start of a file changes only the chunks around it and all later boundaries stay put. LBFS introduced this for file systems (Muthitacharoen et al., SOSP 2001) and FastCDC made the boundary test faster (Xia et al., USENIX ATC 2016).

Fixed blocks are simpler and are enough when edits overwrite in place (photos, databases, office files saved in full). CDC wins when data is inserted, such as logs, text and archives re-packed with small changes. rsync-delta-algorithm (no page yet) is the third option, used when the other side already has an old version and you must find what changed without a prior block list: a weak rolling checksum finds candidate matches at every byte offset, a strong hash confirms them (Tridgell's PhD thesis, the rsync algorithm). It trades CPU for the ability to dedupe against any old copy, so it suits a one-shot transfer between two machines; a sync engine that keeps block lists for every revision gets the same effect from them. Hashes are Cryptographic hashes (SHA-2, BLAKE3): collision-resistant, so the hash can be the block's identity. SHA-256 is the interoperable choice (the API above, Syncthing's device IDs) and BLAKE3 the fast one for a private protocol. The contract: a file is a revision with an ordered list of (offset, size, hash).

Block store

Blocks are immutable and named by hash, which is an ideal fit for Object storage: put-if-absent by key, no update path, any region, and the cost is mostly per GB-month plus requests and egress. Deduplication is then a lookup: before uploading a block, the client asks the server which of the hashes it lacks. Each block is compressed with Zstandard (zstd) (or LZ4 when CPU is tight; Syncthing uses LZ4 for metadata messages per its spec) before encryption, because ciphertext does not compress. Encryption uses Envelope encryption: a data key per block or per file, wrapped by a key held in a key service, so rotation re-wraps small keys instead of rewriting petabytes. Dedup and encryption interact: if the identity of a block is the hash of its plaintext, then equal blocks of different users are one object, which saves space and leaks "this user has this file". restic's design document describes the related attack that recovers the secret chunker parameter from observed chunk sizes (Alexeev, Percival and Zhang, 2025) and its mitigation since 0.18.0, assigning chunks to pack files at random. A sync service therefore dedups within an account or a team, or accepts the leak knowingly.

Transfer

Mobile links and laptops that sleep make interrupted transfers the normal case, so uploads are Resumable uploads: open a session, send offset-addressed pieces, and after any failure ask the server how much it holds and continue from there. Retries use Exponential backoff with jitter, which matters most after a server outage, when millions of clients reconnect at once. Because a block is content-addressed, a repeated upload of the same block is harmless; the commit that follows is the only step that needs an Idempotency key, since it changes metadata.

Namespace and journal

The metadata service holds, for each account or shared folder (a namespace), the tree: file ID, parent ID, name, revision, block list, deleted flag. Writes go through a Write-ahead log (crash-safe, ordered, replayable), and lookups use a B+ tree for point queries and range scans (list a folder). The commit operation is a compare-and-set: "make this block list revision N+1 of file F, if F is currently at revision N" (the pattern of Optimistic concurrency control); if the head moved, the commit fails and the client reconciles. Each journal entry gets a sequence number; that number is the cursor.

Dropbox's public account of its sync engine draws out what the data model must contain: stable IDs that survive a move (the old design stored a move as a delete plus an add, so a network hiccup between the two made files vanish), atomic moves independent of subtree size, and one agreed order for conflicting folder moves (source: Rewriting the heart of our sync engine, Dropbox, 2020).

Cursors

A client keeps one opaque token. Incremental sync cursors return "everything after this token plus the next token", so a sync costs what changed, not what exists; a client that has been away for weeks pulls a long but finite list, not the tree. The token is a position in the journal (a Message sequence number). If the server compacts its journal past the client's token, the client must fall back to a full listing, so compaction horizons have to be longer than the longest device absence you promise to support.

Conflict handling

With a single server journal, ordering is easy: the compare-and-set above gives every file a total order and a conflict is just "your parent is stale". Peer-to-peer systems have no such arbiter, so Syncthing's protocol carries a version vector per file: one counter per device, and comparing two vectors says whether one version descends from the other or they are concurrent (source: Syncthing BEP spec). That is Vector clocks used exactly as designed. For a concurrent edit of file content there is no merge for arbitrary binary data, so the loser is kept as a "conflicted copy" next to the winner; for concurrent tree changes (two moves forming a cycle) the rule has to be written down and tested. CRDTs are the alternative for content that can merge (text, shared documents) and are overkill for the file layer.

Notification

Polling the cursor every few seconds would cost every idle client a request. Instead the client holds one connection and the server pushes "namespace X moved"; the client then pulls with its cursor. The push is one-way and small, so Server-sent events are a good fit: each event carries an ID, and a reconnecting client names the last one so the server resumes. Long polling is the fallback behind proxies that cut streams. The notification carries no data; losing it only delays the next pull, which is why a periodic poll still runs as a safety net.

Local apply

Downloading a file means writing it where the user's editor may also be reading it. The client writes blocks to a temporary file in the same directory, flushes it with the right call (fsync, O_DSYNC and O_DIRECT; on macOS plain fsync does not reach the platter, F_FULLFSYNC does), then renames it over the target, so the file is either entirely old or entirely new after a crash. Before overwriting, the client checks that the local file still matches the state database; if the user edited it meanwhile, that is a conflict, not an update.

Data and control flow

A local edit reaches another device:

1. The watcher reports a path; after a debounce the chunker reads the file and computes the block list.
2. The client sends the list of hashes; the server answers with those it lacks.
3. Missing blocks are compressed, encrypted and uploaded in a resumable session, in parallel for large files.
4. The client commits (file id, parent revision, block list); the server runs the compare-and-set, appends to the journal and returns the new revision.
5. The server pushes a notification to the namespace's other devices.
6. Device B pulls from its cursor, learns the new revision, asks for the blocks it lacks (blocks it has from older revisions or other files are reused), and applies the file through temp file and rename. Time goes to hashing and upload for the first sync of big files, and to the metadata round trip and notification for small edits (order of a second end to end on a good link).

A device returns after weeks:

1. It holds a cursor from the last sync. It pulls the journal tail and merges it with its local state database.
2. Local changes made offline are found by a rescan and committed against the revisions it knew; stale parents are conflicts.
3. Remote changes are downloaded; blocks already on disk are not fetched again.

Scaling

In the order they bite: (1) the client's memory and scan time with millions of files, which Dropbox names as the limit its first engine could not push past (source: Dropbox 2020); levers: a compact state database, lazy hashing, selective or online-only folders. (2) Metadata throughput per namespace: shard by namespace (account or shared folder) so a commit touches one shard; keep rows small by storing block lists, not blocks. (3) Notification fan-out: connections are long-lived and mostly idle; keep them on a tier of cheap connection holders in front of the journal and let a shared folder with thousands of members fan out through a topic. (4) Block storage volume: dedup, compression and tiering of cold blocks in object storage; egress cost grows with downloads, not storage. (5) Upload bandwidth: delta sync and parallel block uploads; large first syncs dominate. (6) Testing: the state space of file states and user actions is astronomical (source: Dropbox 2020), which is why the sync core should be deterministic and tested with Deterministic simulation testing.

Failure modes

Real implementations

The alternative: a plain mirror (rsync, scheduled copy) needs no metadata service and no notifications, but has no history, no conflict model and a full tree scan per run. Choose the sync engine when several people or devices edit the same tree and must see each other's changes in seconds; choose mirroring for one-way backups.

Sources

• Dropbox, Rewriting the heart of our sync engine (2020). post
• Dropbox API documentation, Content Hash. docs
• Syncthing, Block Exchange Protocol v1. spec
• restic, design document (doc/design.rst). repo
• B. Alexeev, C. Percival, Y. X. Zhang, Chunking Attacks on File Backup Services using Content-Defined Chunking, 2025, as cited in the restic design document. eprint
• A. Muthitacharoen, B. Chen, D. Mazières, A low-bandwidth network file system, SOSP 2001. paper
• W. Xia et al., FastCDC: A Fast and Efficient Content-Defined Chunking Approach for Data Deduplication, USENIX ATC 2016. paper
• A. Tridgell, Efficient Algorithms for Sorting and Synchronization (PhD thesis, 1999). pdf
• Seafile README. repo