container-runtime

How it is built

Architecture overview

A container runtime turns an image (a stack of file-system layers plus a configuration) into a running, isolated process on a shared Linux kernel. It has two halves. The image half fetches, verifies and unpacks layers into a local store and assembles a root file system from them by stacking, without copying. The isolation half starts a process with its own view of the machine (namespaces), a resource budget (cgroups) and a reduced set of things it may do (capabilities, seccomp, mounts), then watches it until it exits. The idea that makes it work is that a container is not an object in the kernel but a recipe of ordinary kernel features applied to one process, and that the recipe and the images are standardised by the Open Container Initiative (OCI), so a high-level daemon (containerd), a low-level runtime (runc) and a sandbox runtime (gVisor, Kata Containers) can be swapped behind the same interface. When a shared kernel is too weak a boundary, the same interface starts a user-space kernel or a micro virtual machine instead.

Parts:

• Image format and registry protocol — layers, manifests, digests, how blobs are named and fetched — oci-image-layout (no page yet), tar-archive-format (no page yet), deflate (no page yet), Content-addressed storage, merkle-dag (no page yet), Cryptographic hashes, HTTP range requests
• Image store and snapshots — unpack layers once, share them between containers — overlay-filesystem (no page yet), copy-on-write-snapshots (no page yet), atomic-file-replacement (no page yet)
• Lazy pulling — start before the download finishes — lazy-image-pulling (no page yet), fuse-userspace-filesystem (no page yet), seekable-compressed-stream (no page yet)
• Container setup — namespaces, cgroups, root file system, network devices — Cgroups and namespaces, virtual-network-interfaces (no page yet)
• Isolation hardening — reduce what a compromised process can do — Least privilege and isolation, seccomp-bpf (no page yet), syscall-interception (no page yet), ptrace-process-control (no page yet), ebpf (no page yet)
• Accounting — measure what each container uses — resource-usage-accounting (no page yet)

 registry ──manifest + blobs (HTTPS, digest-addressed)──►┌──────────── node ─────────────────────────────┐
 (OCI distribution)        ▲ range requests (lazy)       │ daemon (containerd-like)                      │
                                                          │   ├ content store: blobs by sha256            │
                                                          │   ├ snapshotter: unpack layer ► snapshot      │
                                                          │   │     (overlayfs, or FUSE for lazy pull)    │
                                                          │   └ task: create ► start ► kill ► delete      │
                                                          │            │ bundle = config.json + rootfs    │
                                                          │            ▼                                  │
                                                          │   low-level runtime (runc-like)               │
                                                          │   clone(namespaces) · cgroup · pivot_root     │
                                                          │   seccomp · drop caps · exec                  │
                                                          │            ▼                                  │
                                                          │   container process  ──syscalls──► host kernel │
                                                          │        or: user-space kernel / micro-VM        │
                                                          └───────────────────────────────────────────────┘

How the parts fit together

Image format and registry protocol

An OCI image is a few small JSON documents and some tar files. The manifest lists a config blob (command, environment, the layer chain) and an ordered list of layer blobs, each named by the SHA-256 digest of its bytes. A registry serves blobs by digest over HTTPS (OCI distribution spec). Because every reference is a hash of the thing it points to, the structure is a merkle-dag (no page yet) and the store is Content-addressed storage: an identical layer used by a hundred images is stored and downloaded once, and a downloaded blob can be verified against the digest in the manifest regardless of which mirror served it. See Cryptographic hashes. The manifest layout is the subject of oci-image-layout (no page yet).

A layer is a changeset, stored as a tar-archive-format (no page yet) archive, usually compressed with deflate (no page yet). Applying a layer means extracting the tar on top of the layers below. Deletion is encoded in the archive itself: an entry named .wh.<name> (a whiteout) means "remove <name> from the lower layers" (OCI image spec, layer). The image config records two hashes per layer: the digest of the compressed blob for download, and the digest of the uncompressed content (the diffID) for identity of the file-system content, so the same content compressed differently is still the same layer to the runtime.

Image store and snapshots

The daemon unpacks each layer once into a snapshot, and a container's root file system is a new snapshot stacked on the top one. On Linux the usual mechanism is an overlay-filesystem (no page yet): a mount with several read-only lowerdirs (the layers), one writable upperdir (the container's changes) and a workdir, presented as one merged directory. A read goes to the highest layer that has the file; a write to a lower-layer file first copies it up into the upper directory; a delete creates a whiteout in the upper directory (kernel overlayfs documentation). This is a file-level case of the idea of copy-on-write-snapshots (no page yet): starting a container costs one mount and an empty directory, not a copy of the image. Block-level alternatives (devicemapper thin pools, btrfs or ZFS snapshots) give the same effect with different costs on small-file workloads.

Unpacking must be crash-safe. A layer is extracted into a temporary directory and then committed by a rename, so an interrupted pull never leaves a half-unpacked snapshot that looks valid; this is atomic-file-replacement (no page yet) applied to directories and to the metadata that names them.

Lazy pulling

Pulling a multi-gigabyte image before starting is the largest part of cold-start time, yet a typical container reads only a fraction of its files at startup. lazy-image-pulling (no page yet) starts the container as soon as the manifest and a small index have arrived, and fetches file contents on first read. Three things are needed:

• Random access into a layer. A normal tar.gz must be read from the start. The eStargz format compresses each file as its own gzip member and appends a table of contents, so a reader can find a file's offset and fetch just that byte range (eStargz specification in the stargz-snapshotter repository). This is a seekable-compressed-stream (no page yet), and it stays a valid OCI layer, so images work with or without lazy-aware tooling.
• Fetching ranges. The registry or blob store is asked for HTTP range requests on the layer blob.
• A file system that fetches. A snapshotter running as a fuse-userspace-filesystem (no page yet) daemon answers the kernel's reads by fetching the needed chunks, caching them, and ideally prefetching the files the image is known to read at startup. The cost is moved, not removed: the first reads of each file wait on the network, so lazy pulling helps most when startup touches little of the image, and a registry in the same region matters.

Container setup

The low-level runtime receives a bundle: a directory with a root file system and a config.json. The OCI runtime spec defines the lifecycle as create (build the environment but do not run the user program), start, kill and delete, with hooks at defined points. Splitting create from start lets the daemon configure networking from outside after the namespaces exist and before the process runs.

The setup itself is Cgroups and namespaces:

1. clone or unshare into new namespaces: mount, PID, network, UTS, IPC, cgroup and optionally user and time (the eight namespace types in namespaces(7)).
2. Join a cgroup and write limits (memory, CPU weight and quota, pids, I/O) into the cgroup v2 files.
3. Mount the overlay as the new root, mount /proc, /dev and bind mounts, then pivot_root into it, so the old file system is no longer reachable by path.
4. Wire the network: the daemon or a plug-in creates a veth pair with one end in the container's network namespace and the other on a bridge or routed on the host (virtual-network-interfaces, no page yet), assigns addresses, and adds NAT or routing rules. The container-networking interface (CNI) standardises this step.
5. Apply hardening (next section), then exec the user program.

A user namespace maps the container's root to an unprivileged host user, so root inside is not root on the host; this is the basis of rootless containers (user_namespaces(7)).

Isolation hardening

Namespaces limit what a process can see; they do not limit which kernel code it can reach. The kernel's system call surface is the shared attack surface, so the runtime applies Least privilege and isolation:

• Drop Linux capabilities to a small set, set no_new_privs, and mount parts of the file system read-only.
• Install a seccomp-bpf (no page yet) filter: a small program run on every system call that allows, denies or traps by number and arguments (seccomp(2)). Default profiles block calls a normal service never needs.
• Use mandatory access control profiles (AppArmor, SELinux) for file and socket policy.

These make a shared-kernel container harder to escape, but a kernel bug in an allowed call is still reachable. Two stronger designs change the boundary:

• User-space kernel (gVisor). The application's syscalls are intercepted (syscall-interception (no page yet), through a pluggable platform: gVisor offers KVM, systrap and ptrace platforms) and served by the Sentry, a kernel written in Go that itself makes only a restricted set of host calls; a Gofer process, with slightly more privilege, mediates file access (gVisor architecture guide). The host kernel sees a narrow, well-filtered interface from one process. The price is overhead on system-call-heavy and I/O-heavy workloads and incomplete Linux compatibility.
• Micro virtual machine (Kata Containers, Firecracker). Each container or pod runs in its own lightweight VM with its own guest kernel; the boundary is the hypervisor. Strongest isolation of the three, with more memory per container and a boot step.

Interception mechanisms are covered in ptrace-process-control (no page yet) and ebpf (no page yet), the latter also used to observe and enforce policy in the host kernel without changing it.

Accounting

The same cgroup tree that limits a container also counts its usage: CPU time, memory (including page cache), I/O bytes, pids. resource-usage-accounting (no page yet) samples these counters into per-container metrics that a scheduler and a billing system use. The memory number that matters for OOM decisions is the cgroup's working set, not the process's resident size.

Data and control flow

Start a container from an image reference:

1. Resolve the tag to a manifest digest; fetch the manifest and config from the registry.
2. For each layer not in the content store, fetch the blob (full, or just the index for lazy pulling), verifying the digest as it streams in.
3. Unpack layers into snapshots, each on top of its parent (one commit per layer; shared layers are already there).
4. Prepare a new writable snapshot for the container and mount the overlay.
5. Write the bundle (config.json from the image config plus runtime overrides) and call the runtime's create.
6. The runtime sets up namespaces, cgroup, root, hardening; the daemon wires the network.
7. start execs the user program; the runtime exits or stays as a small supervisor (a shim) that holds the process's standard streams and exit status. Time goes to: the download (seconds for a large cold image, nearly zero when cached or lazy), the unpack (CPU and disk, mostly decompression and small-file writes), and the setup (milliseconds to tens of milliseconds, order of).

Stop it: kill sends a signal (SIGTERM first); on exit the shim records the status, the runtime's delete removes the cgroup and namespaces, the daemon unmounts and discards the writable snapshot, and read-only layers stay for the next container.

Scaling

In the order each limit bites:

1. Start latency. Cache layers on the node, put a registry or pull-through cache in each region, use lazy pulling, keep images small (fewer, smaller layers, the minimal base).
2. Density. Containers share the kernel and the layer page cache, so memory per container is small; the limits are pids, file descriptors, inode tables, connection tracking and the cgroup tree depth.
3. Unpack cost. Parallel decompression, uncompressed or zstd layers, and sparse snapshot trees.
4. Networking. Per-container veth plus NAT scales to hundreds of containers per node; beyond that, routed or eBPF-based data planes.
5. Isolation strength. Move multi-tenant or untrusted workloads from shared-kernel to a user-space kernel or a micro-VM, and pay the overhead only for those.

Failure modes

Real implementations

For the orchestration layer above the runtime, see Containers and Kubernetes.

Sources

• Open Container Initiative: runtime specification, image specification, distribution specification.
• Linux kernel documentation: overlayfs, control group v2.
• Linux man pages: namespaces(7), user_namespaces(7), seccomp(2), pivot_root(2).
• gVisor: architecture guide, platforms.
• eStargz: specification and lazy pulling design.
• Container Network Interface: containernetworking/cni.