container-build

How it is built

Architecture overview

A container image builder turns a recipe (usually a Dockerfile) and a source tree into an OCI image: a manifest, a config and a stack of layer tarballs, each stored and addressed by the digest of its bytes. The expensive thing is not producing one image but producing the ten-thousandth, where almost nothing changed. So the builder is organised as an incremental build system. A frontend compiles the recipe into a dependency graph of operations; a solver walks that graph, asks a cache whether each node was already computed, and sends only the misses to a sandboxed executor that runs the step on a union filesystem; a layer exporter turns each step's filesystem changes into a compressed tar layer and assembles the image. The idea that holds it together is that every node and every layer is identified by a digest of its inputs or its content, so cache hits, deduplication, remote cache sharing and registry uploads are all the same comparison. BuildKit is the open reference; Kaniko, now archived, and Buildah are the other documented designs.

Parts:

• Frontend — recipe to build graph — layered-image-build (no page yet), built on oci-image-layout (no page yet)
• Solver — schedules graph nodes, deduplicates concurrent work, decides cache hits — DAG workflow execution, incremental-computation (no page yet)
• Cache — digest-keyed results, local and remote — Content-addressed storage, merkle-tree (no page yet), Cryptographic hashes (SHA-2, BLAKE3)
• Executor — runs one step in isolation on a layered filesystem — Cgroups and namespaces, overlay-filesystem (no page yet)
• Layer exporter — filesystem diff to layer blob — tar-archive-format (no page yet), deflate (no page yet), Zstandard (zstd)
• Image assembler — manifest, config and index — oci-image-layout (no page yet), Content-addressed storage
• Reproducibility — same inputs, same digests — reproducible-builds (no page yet)
• Distribution format — layers readable before they are fully downloaded — lazy-image-pulling (no page yet)

 Dockerfile / other recipe
          │  frontend (runs in the builder)
          ▼
   build graph: vertices with digests ───────────────┐
          │                                          │ cache lookup by digest
          ▼                                          ▼
       solver ──miss──► executor (namespaces + cgroups, overlay rootfs)
          │  hit                  │ upperdir = this step's changes
          │                       ▼
          │               layer exporter: diff ─► tar ─► gzip/zstd ─► blob + digest
          ▼                       │
   cache (local + registry/S3) ◄──┘        image assembler: config + manifest ──► registry

How the parts fit together

Frontend: a recipe is a program that emits a graph

BuildKit separates the language from the engine. Its README describes a binary intermediate format, LLB: protobuf messages that define the dependency graph of processes in a build, concurrently executable, efficiently cacheable and independent of Dockerfile syntax; it compares LLB to LLVM IR (source: BuildKit README). Frontends are components that run inside BuildKit and convert a build definition to LLB, so the Dockerfile syntax can be versioned without changing the daemon. This is layered-image-build (no page yet) in its general form: each instruction becomes a vertex whose inputs are the vertices before it, which is why instruction order decides cache reuse (source: Docker docs, build cache).

The contract between frontend and solver is the vertex: an operation definition, a list of input edges, and a digest that covers the definition and all inputs. The image the recipe starts FROM is itself a vertex, resolved to a digest; pinning that digest in a source policy is how BuildKit reproduces pinned dependencies (source: BuildKit docs/build-repro.md).

Solver: a build system over content addresses

The solver is a DAG workflow execution engine, with the twist that the unit is a cache lookup rather than a task. BuildKit's design note says it is heavily optimised for deduplicating work and concurrent requests, so two vertices with the same digest running at once wait on one execution, and that it avoids computing content checksums of snapshots after every command, because that is too slow, doing it only when it can matter (source: BuildKit docs/dev/solver.md). That is incremental-computation (no page yet): re-evaluate a node only when an input changed, and prefer a cheap equality test (definition digest) before an expensive one (content scan).

Two kinds of cache key result. A definition key is a hash of the vertex and its inputs' keys: cheap, and enough for RUN steps, where Docker's docs state the cache check uses only the command string and does not inspect the files the command changed. A content key is a checksum of the input files: required for COPY and ADD, where the cache checksum comes from file metadata and ignores mtime (source: Docker docs, cache invalidation). The solver prefers the cheap match when it can; BuildKit's note describes skipping a content-based check on one input when another input already produced a definition match, so a pull is avoided.

Consequence worth knowing: a RUN apt-get update is cached forever by its command string, so rebuilding a week later gives the same packages. The cache is a statement about inputs, not about the world.

Cache: digests for results, local and remote

Results are stored by Content-addressed storage: a layer is a blob named by its SHA-256, the cache record maps a key to those blobs, and any builder that can read the store can reuse them. The hash is a Cryptographic hash because keys are also trusted for deduplication and for registry verification. An image is a merkle-tree-shaped structure (no page yet): the manifest names the config and layer digests, the index names manifests, so equal subtrees are equal bytes and an unchanged layer never crosses the network again (the OCI spec describes the subject relation as an association with a Merkle DAG).

BuildKit exports cache in several forms: inline (cache metadata embedded in the image config, which only supports the min mode), a separate registry cache that can use max mode, a local directory, and experimental GitHub Actions, S3 and Azure Blob backends (source: BuildKit README). The choice is a CI trade-off: inline cache costs nothing extra to publish but caches less; a separate cache artifact stores intermediate layers too. For a fleet of builders without a shared cache store, the README suggests client-side load balancing by consistent hashing so a given graph keeps landing on the daemon that has its cache.

Executor: namespaces, cgroups and a union rootfs

Each non-cached step runs a process in a fresh container: Cgroups and namespaces limit what it sees (own mount table, PID space, network) and what it may use. The root filesystem is a union of the previous layers (read-only) plus a writable layer, which is the job of overlay-filesystem (no page yet). The kernel documentation describes the mount as lowerdir, upperdir and a workdir on the same filesystem as the upper (source: Linux kernel docs, overlayfs). The write layer of the step is exactly the diff the exporter needs.

Two contracts matter. First, the executor is the trust boundary: a Dockerfile is arbitrary code, and a builder that accepts recipes from untrusted users must not give them host root. BuildKit supports running its daemon without root, with the stated limits that the overlayfs snapshotter needs kernel 5.11 or an Ubuntu kernel (otherwise fuse-overlayfs or the slower native snapshotter is used) and that network mode is host (source: BuildKit docs/rootless.md). Second, Kaniko took a different line: it executes each Dockerfile command in userspace inside a container, snapshotting the filesystem itself, so it needs no daemon (source: Kaniko README). The project is archived since 2025.

Layer exporter: diff, tar, compress

A layer is the changeset between two filesystem states, serialised as a tar archive. The OCI layer spec defines the media types application/vnd.oci.image.layer.v1.tar, +gzip and +zstd, and represents deletions with "whiteout" files: an empty file named .wh. plus the basename to remove, or .wh..wh..opq to hide all children of a directory (source: OCI image spec, layer). The overlay kernel format differs (a whiteout is a character device with device number 0/0, or a zero-size file with an extension attribute; source: Linux kernel docs), so the exporter converts between the two. tar-archive-format (no page yet) matters because the archive's bytes are the layer's digest: entry order, mtimes, uid/gid and header format all change the hash.

Compression is a choice, not a default. deflate (no page yet) in gzip form is universally supported and BuildKit's default; Zstandard covers a wider speed and ratio range and is allowed by the OCI media types; BuildKit's exporter accepts compression=uncompressed|gzip|estargz|zstd (source: BuildKit README). Two digests result and must not be confused: the layer descriptor digest covers the compressed blob; the config's rootfs.diff_ids hold the digest of the uncompressed tar (the DiffID), and the ChainID hashes the applied stack (source: OCI image spec, config). Changing the compression changes the blob digest but not the DiffID.

Image assembler

The assembler writes the config blob (environment, entrypoint, history, DiffIDs), then the manifest listing the config and layer descriptors, then an index when several platforms were built. All are content addressed, so the output is a set of blobs plus one top-level digest, which can be written as an OCI layout directory, a tar, or pushed (BuildKit outputs include image/registry, local directory, Docker tarball and OCI tarball; source: BuildKit README). When pushing, the registry's blob-existence check turns unchanged layers into no-ops; see the Artifact registry system page.

Reproducibility

reproducible-builds (no page yet) asks that identical inputs produce identical bytes, which for images means identical layer digests. The sources of difference are known: file timestamps, entry order, uid/gid, absolute paths, and unpinned inputs. BuildKit supports SOURCE_DATE_EPOCH as a build argument (the value context takes the commit time of a git context), and the exporter option rewrite-timestamp=true rewrites file timestamps to that value (source: BuildKit README and docs/build-repro.md). The tension with caching is real: Docker's docs warn that setting SOURCE_DATE_EPOCH to a commit timestamp invalidates the cache for WORKDIR and everything after it on every commit, so a fixed value is recommended when cache matters more than provenance.

Distribution format for lazy pulling

The output format can make the deploy side faster. lazy-image-pulling (no page yet) starts a container before the image is fully downloaded and fetches chunks on demand with HTTP range requests; for that the layer must be seekable and each file independently verifiable, which ordinary tar+gzip is not. eStargz is a backward-compatible gzip layer format that adds a table of contents, per-chunk digests and a prefetch list, and stargz-snapshotter reads it through FUSE (source: stargz-snapshotter docs). The builder's job is only to emit it (compression=estargz with OCI media types, per the BuildKit README); the cost is a larger build step and a bit of extra size.

Data and control flow

Rebuild after a one-file change.

1. The frontend parses the Dockerfile and emits the graph; each vertex digest is computed from its definition and inputs.
2. The solver asks the cache, vertex by vertex from the base image forward: image vertex (digest), RUN steps (definition key), COPY (content key over the copied files, mtime ignored).
3. The first vertex whose key misses starts execution; all later vertices miss too, because their input changed.
4. The executor mounts the previous layers read-only plus a fresh upper layer and runs the step in its sandbox.
5. The exporter tars the upper layer (sorted, normalised if reproducibility is requested), compresses it, hashes both forms, stores the blob and records the cache entry.
6. The assembler writes config and manifest; the push sends only blobs the registry does not already have.

Build with a remote cache in CI.

1. A fresh builder imports cache metadata for the target reference (type=registry).
2. The solver matches vertex keys against the imported cache; BuildKit's design note says it loads cached layers only for the final matching node, not for every vertex.
3. Matched layers are pulled on demand; misses are built; with max cache mode all intermediate layers are exported.

Scaling

1. Cache hit rate decides almost everything. Lever: order instructions from least to most frequently changed, use cache mounts for package managers, and share a remote cache between CI workers.
2. Step latency on a miss is bound by the step itself (a compile or a package install). Lever: more parallelism in the graph (multi-stage builds are independent branches the solver runs concurrently), bigger machines.
3. Layer I/O: tar, hash and compress are CPU and disk bound. Lever: zstd, fewer and larger layers, avoiding content scans (the solver's laziness).
4. Cache and registry storage grows without bound. Lever: prune policies (BuildKit has automatic garbage collection configured through buildkitd.toml), retention rules in the registry.
5. Many builders: a shared registry cache, or consistent hashing of builds onto daemons.
6. Deploy latency is a separate scaling axis solved by the output format: lazy pulling and layer deduplication.

Failure modes

Real implementations

The alternative: build the image from a language-native build graph (Bazel-style rules or Nix) that never runs a Dockerfile; you get stronger reproducibility and finer layers at the price of describing the build in that system's terms. Pick the Dockerfile route for ecosystem reach and speed of adoption.

Sources

• BuildKit: README, solver design, build reproducibility, rootless mode
• Docker docs, build cache: invalidation
• OCI Image Specification: layer, config, manifest
• Linux kernel documentation, Overlay Filesystem
• stargz-snapshotter: README and eStargz format
• Kaniko: README
• Reproducible Builds project: SOURCE_DATE_EPOCH, archives