image-snapshotter

How it is built

Architecture overview

A container image is a stack of read-only layers plus a small config, all stored as blobs under the hash of their bytes. A container runtime has to turn that stack into one mountable root filesystem, give each running container a private writable layer on top, and do so quickly and many times per node. The component that does this is the snapshotter: it keeps a tree of filesystem snapshots, one per layer, and hands the runtime a list of mounts for any of them. The default design unpacks every layer into a directory once and then stacks the directories with a union mount (overlayfs). The main alternative design never unpacks the layer at all: it mounts the layer straight from the registry and fetches only the bytes the container touches (lazy pulling).

The one idea is the split between content and snapshots. Content is immutable, verified by digest and shared by every image that contains it. Snapshots are cheap, mutable views derived from content, created and destroyed with the container. Lazy pulling, block-device images and VM-friendly snapshots are other ways to implement the snapshot half behind one small interface. This page follows containerd, whose snapshotter interface is the common plug-in point.

Parts:

• Content store — immutable blobs (manifests, configs, layers) addressed by digest, with reference-based garbage collection — Content-addressed storage, Cryptographic hashes, zstd
• Layer format and unpacker — turns a compressed tar changeset into a snapshot, applying whiteouts for deleted files — tar-archive-format (no page yet), merkle-dag (no page yet), zstd
• Snapshotter core — the Prepare / View / Commit / Remove / Mounts tree of snapshots; the default backend stacks directories with a union mount — overlay-filesystem (no page yet), copy-on-write-snapshots (no page yet), atomic-file-replacement (no page yet)
• Lazy-pull path — mounts a layer from the registry and fetches ranges on demand, verifying each chunk — seekable-compressed-stream (no page yet), HTTP range requests, fuse-userspace-filesystem (no page yet), merkle-tree (no page yet), content-defined-chunking (no page yet)
• Runtime isolation — what the container sees and may use once its rootfs is mounted — Cgroups and namespaces, mmap and the page cache, Virtual memory and the TLB
• Trust — signatures over manifests so a node runs only the image its owner published — Ed25519 signatures

 registry ──manifest, config, layer blobs (by digest)──► content store (sha256/<digest>)
                                                              │ unpack, layer by layer (tar + whiteouts)
                                                              ▼
   snapshots:   [layer0] ◄── [layer1] ◄── [layer2]  (committed, read-only, named by chainID)
                                              ▲
                              Prepare(key, parent=layer2)
                                              │
                          active snapshot (writable "upper" dir)
                                              │ Mounts() → overlay: lowerdir=layer2:layer1:layer0, upperdir=…
                                              ▼
                                  container rootfs  (namespaces + cgroups)

   lazy path:  registry ◄── HTTP Range ── FUSE/EROFS mount of the layer ── [layer1 "remote snapshot"]

How the parts fit together

Content store and image format

An image is a manifest listing the digest of a config blob and of each layer blob, optionally reached through an index that picks one manifest per platform. The runtime stores each blob under its digest, so images sharing a layer share the bytes and a pull skips blobs already present. That is Content-addressed storage with SHA-256 (Cryptographic hashes): a blob from disk or a mirror can be checked without trusting the transport. Layers are tar archives compressed with gzip or, in the OCI media types application/vnd.oci.image.layer.v1.tar+gzip and ...tar+zstd, with zstd (source: OCI image-spec layer.md). The image is a Merkle DAG (merkle-dag, no page yet): a signature over the manifest digest covers every byte below it.

Two digests per layer matter. The manifest names the compressed blob; the config's rootfs.diff_ids lists the digest of each uncompressed changeset (the DiffID). The ChainID of a layer stack is Digest(ChainID(parent) + " " + DiffID(layer)) (source: image-spec config.md), so it identifies the layer and everything below it. Snapshotters name committed snapshots by ChainID: that is the contract between content store and snapshotter, so two images with the same layer prefix reach the same snapshot and unpack it once.

containerd garbage-collects by reachability: a manifest carries labels such as containerd.io/gc.ref.content.config and containerd.io/gc.ref.content.l.<index> that keep its children alive, and a config carries a label that pins its snapshot (source: containerd docs/content-flow.md). This is Mark-sweep garbage collection over the metadata graph, not over a heap. Deleting an image frees only what no other image reaches.

Layer format and unpacker

A layer is a changeset against the layers below it (source: image-spec layer.md). Added and modified files are stored in full; a deleted file is an empty whiteout entry named .wh.<name>, and an opaque whiteout hides everything below in one directory. The unpacker reads the tar stream, Prepares a snapshot, writes the entries into it and Commits it under the ChainID.

Tar has no index: to read one file the stream up to it must be decompressed, and no per-file digests exist inside it. Every lazy-pull format below answers that. Header details (ustar, pax extended headers) live in tar-archive-format (no page yet).

Snapshotter core

The interface is small. A snapshot is either committed (read-only, has a name and a parent), active (writable, has a key and a parent) or a view (read-only active). The calls are Prepare(key, parent), View(key, parent), Commit(name, key), Remove(key), Mounts(key), plus Stat, Update, Usage and Walk (source: containerd core/snapshots/snapshotter.go). The snapshotter returns mounts, a list of {type, source, options} that the runtime executes: it decides how the filesystem is built, the runtime only mounts what it is given.

The default backend, overlayfs, is the same mechanism as Docker's overlay2 driver (source: containerd docs/snapshotters/README.md). Each committed layer is a host directory. A container's mount is one overlayfs mount (overlay-filesystem, no page yet): lowerdir= lists the layer directories top to bottom, upperdir= is the container's writable directory, workdir= a scratch directory on the same filesystem. Reads fall through the stack; the first write to a file copies it up (file granularity; the optional metacopy mode copies only metadata on chown/chmod); a delete creates a whiteout, a character device with number 0/0 (source: kernel overlayfs documentation).

Why a union mount and not the alternatives:

• native copies the parent tree for every snapshot: no kernel feature needed, cost proportional to image size per container.
• btrfs and zfs use the filesystem's own copy-on-write snapshots (copy-on-write-snapshots, no page yet): cheap clones, but that filesystem must sit under the data root.
• devmapper and blockfile give each container a block device or image file, which VM-based runtimes can attach directly (source: containerd snapshotter README).
• erofs is a read-only compressed filesystem for the layers, combined with overlayfs for the writable part.

Crash safety belongs to the snapshotter too: a crash between the filesystem work and the metadata commit must leave either the old state or something detectable as incomplete. That needs atomic-file-replacement (no page yet) and ordered syncs (fsync and direct I/O). overlayfs offers fsync=volatile for upper directories that may be lost in a crash, which suits throw-away container layers (source: kernel overlayfs documentation).

Lazy-pull path

The motivating measurement: pulling packages accounted for 76% of container start time while only 6.4% of that data was read (Harter et al., FAST 2016). A node can start after fetching a small part of the image, if it can fetch pieces.

Three ingredients are needed, and each is a block:

1. A seekable layer (seekable-compressed-stream, no page yet). eStargz compresses each non-empty file as its own gzip member, ends the tar with a JSON TOC (stargz.index.json) and adds a footer that locates it; the result is still a valid layer, so runtimes that know nothing about eStargz can run it (source: stargz-snapshotter docs/estargz.md). Digests per chunk let each fetched piece be verified alone, a Merkle-style layout (merkle-tree, no page yet) where the manifest digest anchors the TOC and the TOC anchors the chunks.
2. A way to fetch bytes: HTTP range requests against the registry blob, first for the TOC, then for the ranges of the files the container opens. eStargz marks prioritized files with a landmark entry so one ranged request prefetches what a start is likely to need.
3. A way to present the files. The stargz snapshotter mounts the layer with FUSE (fuse-userspace-filesystem, no page yet) and offers a separate "fuse manager" process so that restarting the snapshotter does not kill the mounts of running containers (source: docs/overview.md).

The seam with containerd is the remote snapshot protocol: during unpack, containerd calls Prepare with a label carrying the target ChainID; a snapshotter that can serve the layer remotely commits a snapshot of that name and returns "already exists", and containerd skips the download (source: LabelSnapshotRef in snapshotter.go). To the rest of the stack a lazy layer is an ordinary committed snapshot.

The variants differ in where the index lives:

• eStargz rewrites the layer at build time, so the image digest changes.
• SOCI leaves the OCI image untouched and publishes a separate SOCI index artifact next to it in the registry. The index holds a zTOC per layer: a table of contents plus checkpoints of the decompressor state, so any span of the gzip stream decompresses independently (source: soci-snapshotter README, docs/glossary.md). Existing image signatures stay valid.
• Nydus converts to its RAFS format with chunk-level deduplication across layers and images (source: nydus README); content-defined-chunking (no page yet) is the block for cut points that survive insertions.
• OverlayBD exposes layers as block devices through a user-space TCMU backstore, so any filesystem can sit on top and writes can go to a block-level layer (source: accelerated-container-image README).

Lazy pulling trades a cold-start win for a runtime dependency: a first access costs a network round trip, and a lost registry connection can surface as an I/O error in a running container. The mitigations are prefetch lists, a local chunk cache and a background full fetch.

Runtime isolation and the page cache

Once the rootfs is mounted, the container is an ordinary process in new namespaces under a cgroup (Cgroups and namespaces); the snapshotter's job ends at the mount list. Shared lower-layer files are normally read through the host page cache, so containers of one image share cached pages until a file is copied up. Executables and libraries are mapped by virtual memory and loaded by page faults, which is why a lazily mounted layer feels normal: the kernel already reads binaries on demand.

Trust

The digests prove the bytes match the manifest, not who wrote the manifest. A signature over the manifest digest closes that gap (Ed25519 signatures, or ECDSA). Signing the digest is enough because the hash structure underneath fixes every layer and chunk. It is also why SOCI keeps the original image: a scheme that rewrites layers invalidates signatures made over the original.

Data and control flow

Pulling and starting a container with the default snapshotter:

1. The runtime resolves the tag to a digest, downloads the index and platform manifest, then the config and the layer blobs missing from the content store. Each blob is hashed as it is written and rejected on mismatch. Network-bound.
2. For layer n the unpacker computes the ChainID, calls Prepare(extract-…, parent=chain(n-1)) with ChainID and DiffID as labels, decompresses the blob, applies the tar entries (whiteouts delete, opaque markers hide) and Commits. CPU- and disk-bound; layers can be fetched in parallel, each is decompressed as one stream.
3. Prepare(container-key, parent=chain(top)) creates the writable layer and returns the overlay mount; the runtime mounts it in the container's mount namespace and starts the process.
4. On exit Remove(container-key) deletes the upper directory; committed layers stay for the next container.

Starting a container with a lazy-pull snapshotter:

1. The runtime pulls manifest and config only. For each layer the snapshotter reads its TOC with a ranged request (the tail of an eStargz blob, or the SOCI zTOC), mounts the layer by FUSE and answers "already exists" to the unpacker.
2. If the image has a prefetch range, one request fetches it; the overlay mount is returned with the remote layer directories as lower layers.
3. Each first read of a file becomes a range request, a digest check against the TOC and a copy into a local cache; later reads hit the cache.
4. A background task fetches the rest, so the image eventually behaves like a local one.

Scaling

In the order the limits bite:

1. Cold-start time is dominated by bytes moved before the first instruction runs. Levers: smaller and fewer layers, zstd instead of gzip (decodes at order of 1-2 GB/s), parallel fetch, then lazy pulling, then a cluster cache or peer-to-peer distribution (Nydus lists a P2P backend).
2. Registry bandwidth and request rate at fleet scale: every node pulls the same layers. Levers: a regional mirror or cache, base images baked into the node image, range requests for only the needed chunks.
3. Node disk: layers are shared, but writable layers and unpruned images accumulate. Levers: image garbage collection, per-snapshot Usage accounting, size limits where the backend honours them, a volatile upper layer for scratch containers.
4. Deep stacks: every lower layer is one more directory to search on a lookup. Levers: fewer layers, squashing hot images, a flattened read-only EROFS image.

Failure modes

Real implementations

Sources

• OCI Image Format Specification: layer changesets, whiteouts, config and ChainID. layer.md · config.md
• containerd, Snapshotters and Content flow documents, and the Go interface core/snapshots/snapshotter.go. snapshotters · content-flow
• Linux kernel documentation, Overlay Filesystem. docs.kernel.org
• T. Harter, B. Salmon, R. Liu, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, Slacker: Fast Distribution with Lazy Docker Containers, FAST 2016. paper page
• stargz-snapshotter, eStargz: Standard-Compatible Extension to Container Image Layers for Lazy Pulling and Overview. estargz.md · overview.md
• SOCI snapshotter README and glossary. repo
• Nydus README and accelerated-container-image README, which cites H. Li et al., DADI: Block-Level Image Service for Agile and Elastic Application Deployment, USENIX ATC 2020. paper page