web-archive

How it is built

Architecture overview

A web archive keeps what a web page looked like at a given moment and lets a visitor browse it later as if it were still live. It has two halves that meet in a file format. The capture half is a crawler that fetches pages politely, decides what is new, and appends the raw HTTP exchange to large container files (WARC). The access half looks a URL and a date up in an index, reads exactly one record out of a container file, and rewrites the page so that every link, image and script is fetched from the archive instead of the live web. The one idea that makes it work is that the container file is the source of truth and everything else is rebuilt from it: the index, the full-text search, the replay views and even the crawl statistics are derived data that can be regenerated from the WARC files, so the expensive thing to protect is a pile of immutable, checksummed, compressed records.

Parts:

• Frontier and scheduler — which URL to fetch next, one polite fetch per host at a time — url-frontier (no page yet), built on d-ary heap, token bucket; URL normalization; Robots exclusion protocol; Bloom filter
• Fetchers — download the response with the exact request and response bytes kept — see the Web crawler system page
• Writer — appends records to size-capped container files and hashes every payload — warc-format (no page yet); deflate (no page yet); Cryptographic hashes (SHA-2, BLAKE3)
• Capture index — one sorted line per capture, for lookups by URL, domain and date — sstable (no page yet); prefix-compression (no page yet)
• Replay — finds the closest capture and makes the page work offline from the archive — html-rewriting (no page yet)
• Durable store — keeps the container files for decades on cheap disks — erasure-coding (no page yet), built on Reed-Solomon codes; data-scrubbing (no page yet)

 seeds ─► frontier ─► fetchers ─► WARC writer ──► container files (1 GB-ish, one gzip member per record)
            ▲  │ robots, per-host delay    │              │ erasure-coded store + scrub
            │  └── seen-URL filter ◄───────┤              ▼
            └──── outlinks ◄──────────────┘      indexer ─► sorted capture index (CDX / CDXJ)
                                                              │ url key + timestamp ─► file, offset, length
 browser ─► replay server ─► index lookup ─► read one record ─► rewrite URLs ─► page "as of" date

Capture is write-heavy and limited by how fast sites let you fetch; replay is read-mostly and latency-bound.

How the parts fit together

Frontier and scheduler

The frontier holds the URLs still to be fetched; the Web crawler system page covers a general one in depth. Here it is built from five blocks.

url-frontier (no page yet) is the queue itself: prioritised per-host queues with politeness delays. The scheduler keeps one queue per host and lets only one URI per host be processed at a time (Heritrix documents this rule, with delayFactor, minDelayMs and maxDelayMs for the wait). It hands a fetcher the next URL of a host whose delay has expired, so hosts that are ready wait in a priority queue keyed by their next allowed time, a job for a d-ary heap (a wider heap, fewer levels per operation than a binary one). The same rule can be written as a token bucket per host (rate r, burst b = 1); the bucket is the better fit if a host later allows bursts. A plain FIFO queue is the alternative and it breaks politeness: a large site would fill the head of the queue and the crawler would hit it in a row. Each queue entry carries a host, a priority and a not-before time.

URL normalization puts every URL into a canonical form before it is compared or stored, because the seen-URL test and the index key both depend on it. It hands the same key to the frontier, the seen set and the index; the original string is kept beside it. Without it one page appears under many names, and a rule changed later makes old and new captures of one page look like different pages.

Robots exclusion (RFC 9309) decides per URL whether the crawler may fetch it. The result is cached per host and checked before a URL is released from the frontier. Heritrix has a configurable robotsPolicyName with values obey, robotsTxtOnly and ignore; an archive chooses the policy per crawl and should write it into the crawl's metadata so that later readers know what was skipped.

Bloom filter answers "was this URL queued before" in a few bits per URL, and deletes are impossible, which is fine because the seen set only grows. Its false positives lose a few pages, which a legal-deposit crawl may not tolerate; such archives use an exact on-disk set. The filter sits between link extraction and the frontier: only URLs it has not seen are queued.

Unlike a search crawler, an archive wants to fetch the same URL again: the frontier has a revisit schedule, and duplicates are removed after fetching (see the writer), not before.

Fetchers and the writer

A fetcher keeps the complete HTTP request and response, headers included, because replay needs content type, redirects and character set as much as the body. The writer wraps each exchange as a record of the WARC format (warc-format, no page yet), the container defined by the WARC 1.1 standard (the successor of the ARC format the Internet Archive has used since 1996):

• a response record holds the full HTTP response and a request record the request that produced it, linked by WARC-Concurrent-To;
• a metadata record holds derived facts (outlinks, language); a warcinfo record describes the crawl;
• a revisit record says "this fetch returned the same payload as an earlier capture" and carries the payload digest and a reference instead of the bytes (profile identical-payload-digest), so a page fetched daily for a year costs one body plus 365 small headers;
• WARC-Block-Digest and WARC-Payload-Digest carry a labelled hash (the specification's examples use sha1:), so a record can be verified without trusting the storage.

The writer's contract: append-only, one file per writer, rolled over at a size limit. The standard recommends about 1 GB per file and the Internet Archive's naming convention Prefix-Timestamp-Serial-Crawlhost.warc.gz. Records are compressed one gzip member per record (RFC 1952 allows concatenated members), so a record can be read without decompressing everything before it (deflate, no page yet). The format itself defines no compression; a Zstandard-based variant (.warc.zst, with optional dictionaries) exists as an experimental IIPC specification, whose own text warns that the format is much younger than gzip. For hashes, the choice between a legacy digest and SHA-2 or BLAKE3 trades what old tools verify against resistance to deliberate collisions; the label in the field (sha1:, sha256:) lets both coexist.

Deduplication happens in the writer, not in the frontier: the writer looks up the payload digest in a store of earlier digests, then writes a full response or a revisit record. pywb's recorder offers the three policies a writer needs: write the duplicate, write a revisit record, or write nothing (digest index in Redis). Browser-driven crawlers such as Brozzler capture pages built by JavaScript together with the requests they make.

Capture index

WARC files do not index themselves; pywb states that "the WARC format itself does not provide a specific index", so an external index lists every capture. The classic CDX index has one text line per capture. The Internet Archive's standard 11-field header is CDX N b a m s k r M S V g, giving a massaged URL key, a 14-digit timestamp, original URL, media type, status code, digest, redirect, meta tags, compressed size, offset and file name. The leading key is a SURT (Sort-friendly URI Reordering Transform): the host is reversed, so example.com/ becomes com,example,)/, which puts all subdomains of a domain next to each other. Because lines are sorted by key and then timestamp, one sorted file answers three query types with a binary search and a scan: an exact URL with all its dates, everything under a path prefix, and everything under a domain (matchType=domain in the Wayback CDX server, available only if the index is in SURT order).

This is the structure of sstable (no page yet), the sorted string table of an LSM database; a general database holding one row per capture would add write-ahead logging and random updates that an append-only archive never needs. The index is: a sorted, immutable file, built in bulk by a sort job over the WARC files, merged by sort-merge, queried by seeking. The CDXJ variant used by pywb keeps the sort key (urlkey timestamp) and puts the other fields in a JSON object, so indexes with different field sets can still be merged and sorted. For very large archives pywb documents the ZipNum layout: CDXJ lines are gzipped in groups (often 3,000 lines), concatenated into shard files, and a small secondary index with the first key of each group is binary-searched; each lookup pays for reading and decompressing one group. Neighbouring lines share long URL prefixes, so a group compresses well (prefix-compression (no page yet) is the same idea done explicitly: store each key as the length it shares with the previous key plus the new tail). The contract with replay is small: given a SURT key and a timestamp, return lines with file name, offset and length.

The CDX server API exposes the index with matchType (exact, prefix, host, domain), from/to, filter, collapse (for example one capture per hour with collapse=timestamp:10) and a resumeKey, because offset paging rereads from the start on big queries.

Replay

Replay turns a request (a URL and a date) into a page. Four steps, each a contract with a neighbour:

1. Lookup: the closest capture to the requested date. The Memento protocol (RFC 7089) standardises this: a TimeGate takes a URL plus an Accept-Datetime header and returns the best capture; a TimeMap lists all captures of a URL. pywb implements both.
2. Fetch: the index line gives file name, offset and length; a byte-range read returns one gzip member, which decompresses to one record.
3. Rewrite: the archived page's links point at the live web, which may be gone or changed. The replay layer rewrites URLs in HTML, CSS and headers to an archival URL, for pywb <host>/<collection>/<timestamp><modifier>/<url> (html-rewriting, no page yet). JavaScript is handled mostly on the client: a script injected into the page (wombat, a separate module since pywb 2.3.0) intercepts URL-forming calls at run time, because rewriting JavaScript source statically is fragile. The modifiers choose the treatment: id_ returns the original bytes unchanged, mp_ is the main page rewrite, js_ and cs_ hint that a resource is script or stylesheet.
4. Frame: pywb recommends framed replay (page in an iframe, archive banner outside), partly because a hostile archived page could otherwise tamper with the banner.

Original caching and cookie headers must be neutralised on replay (pywb prefixes some with X-Orig-Archive-).

The rewriting itself is html-rewriting (no page yet): a streaming tokenizer that changes only URL-bearing attributes and passes the rest through. The alternative, parsing to a DOM and serialising it again, would normalise the markup and so change what the archive shows. The contract: replay hands it the record's body, the archival URL prefix and the capture timestamp, and gets back bytes to send to the browser; it never calls the index itself.

Durable store

The container files are the asset: written once, read rarely, kept long, which suits cheap disks organised as a store of large immutable files. The writer hands over closed files; replay asks for a file name plus a byte range. Two blocks keep the bytes alive:

• Erasure coding (erasure-coding, no page yet) is built on Reed-Solomon codes for the math. It stores k data chunks plus m parity chunks. The Ceph documentation gives the arithmetic: the overhead is (k+m)/k, so a 4+2 profile uses 1.5 GiB per GiB of data, against 3.0 for three-way replication, at the cost of slower recovery and rewrites. Archive files are immutable, so the usual penalty (small overwrites) never occurs.
• Scrubbing (data-scrubbing, no page yet) reads everything periodically and compares checksums; a bad chunk it finds is rebuilt from the other chunks of the stripe, so the two blocks need each other. Ceph has light scrubs (sizes and attributes, usually daily) and deep scrubs (read data, check checksums); the archive adds re-hashing against the WARC digests, which survive a change of storage system.

Data and control flow

One capture (crawl to index):

1. The frontier hands a fetcher the next ready URL of a host whose delay has expired; robots rules for the host are cached and checked first.
2. The fetcher downloads the response; the outlinks are extracted, normalised and filtered through the seen-URL set and the crawl scope, then queued.
3. The writer computes the payload digest, looks it up in the recent-digest store and writes either a response record or a revisit record, plus the request record, into the current WARC file; the file rolls over at the size limit.
4. Closed files are copied to the durable store; an indexer reads each file, emits one CDX(J) line per record and merges the lines into the sorted index (daily or per crawl, in batch).

One page view (replay):

1. The browser asks for <collection>/<timestamp>/<url>; the replay server normalises the URL into the SURT key.
2. It queries the capture index for the closest timestamp (secondary-index binary search, one group decompressed).
3. If the line is a revisit, the server follows the digest or reference to the original capture's record.
4. It reads the byte range, decompresses the member, parses the HTTP response, rewrites the body and headers, and returns it.
5. Each embedded resource takes the same path with the same timestamp.

Scaling

In the order the limits bite:

1. Capture throughput is bounded by politeness, not bandwidth: one fetch at a time per host with a delay means a large site takes days. Levers: many hosts in parallel, prioritising which to deepen.
2. The seen-URL set and the frontier grow with discovered URLs. Levers: a Bloom filter or on-disk set partitioned by host, frontier checkpoints (Heritrix has them), scope rules.
3. Storage is the permanent cost. Levers, in order: revisit records for duplicates, better compression (a larger window or a dictionary), then erasure coding in place of replication.
4. Index size and lookup cost: one sorted file is binary-searched; beyond that, shard by key range, gzip line groups (ZipNum), keep the secondary index in memory, and build shards in parallel (pywb: one reduce task per shard).
5. Replay latency is bounded by lookups per page. Levers: cache hot index groups and records, cache rewritten results.

Failure modes

Real implementations

The alternative: for a handful of pages, a "save this page" tool that stores one HTML or PDF file per capture needs no index or rewriting, but it loses the HTTP exchange, so it cannot be verified, deduplicated or replayed with its original behaviour.

Sources

• ISO 28500, The WARC Format 1.1 (IIPC copy, with Annex C file naming and Annex D compression, and the revisit profiles). spec
• IIPC, Zstandard Compression for WARC Files 1.0 (experimental). spec
• IIPC, The CDX File Format (2015), with the 11-field legend CDX N b a m s k r M S V g. spec
• Internet Archive, Wayback CDX Server API: match types, collapsing, limits, resume keys. README
• pywb documentation: Indexing (CDX, CDXJ, ZipNum), Rewriter, Recorder, Warcserver, Memento API, Configuring (framed replay).
• Heritrix 3: configuring jobs, glossary (SURT, frontier).
• H. Van de Sompel et al., RFC 7089, HTTP Framework for Time-Based Access to Resource States: Memento. RFC
• Ceph documentation: erasure code (overhead (k+m)/k), OSD configuration: scrubbing.