tensor-runtime

How it is built

Architecture overview

A portable tensor runtime loads a trained model from a file and executes it on whatever hardware is at hand (server CPU, GPU, phone, browser) behind one API. It is an interpreter for a graph of tensor operators with a compiler's optimizer in front of it. At load time it parses the model, infers shapes, rewrites the graph (constant folding, operator fusion, layout changes), splits the graph among hardware back ends, and plans the memory of every intermediate tensor. At run time an executor walks the nodes in dependency order and calls one kernel per node, or per fused node, on a thread pool or a GPU stream.

The idea that makes it work is a stable operator contract plus replaceable back ends. The model format fixes the meaning of each operator; each back end declares which operators it can run and supplies its own kernels; the runtime assigns each node to the best back end that claims it and falls back to a reference CPU kernel for the rest. A model exported once therefore runs everywhere, and a new chip only needs a back end, not a new model. ONNX Runtime is the widest open example of this design; its back ends are called execution providers.

Parts:

• Model loader — reads the file, builds the in-memory graph, maps weights without copying — Model file formats (no page yet), Protobuf wire format (no page yet), mmap and the page cache
• Graph analysis — shapes, dependencies, constants — Shape inference (no page yet), Dataflow analysis
• Graph optimizer — fusion and layout rewrites, run before and after partitioning — Kernel fusion, Tensor layout transforms (no page yet)
• Partitioner — assigns nodes to back ends and cuts the graph into subgraphs — Operator graph partitioning (no page yet)
• Memory planner — lifetimes to offsets in one arena — Tensor memory planning (no page yet), Arena allocator
• Executor — runs nodes in dependency order, inside and across operators — Topological sort, Work-stealing deque
• Kernel library — the per-back-end operator implementations — Tiled matrix multiplication, SIMD programming, GPU compute shaders, Weight quantization
• GPU launch layer — replays a recorded kernel sequence with one launch — CUDA graphs (no page yet)

 model.onnx / .ort / .gguf
        │ parse (protobuf or flat binary), weights mmap'd
        ▼
  in-memory graph ──► shape inference ──► basic rewrites (fold constants, fuse, drop no-ops)
        │
        ▼ partition: each node → first back end that claims it (GPU, NPU, … else CPU)
  subgraph A (GPU) │ subgraph B (CPU) │ subgraph C (NPU)
        │             extended / layout rewrites, per back end
        ▼
  memory plan: lifetimes → offsets in one arena per device
        ▼
  executor: topological order ──► kernel per node ──► intra-op thread pool / GPU stream
        ▲                                │
  inputs ─────────────────────────────────┴──► outputs     (session reused across requests)

Everything left of the executor is paid once per session; everything right of it is paid per request. The design question at each seam is how much work can move from the second group to the first.

How the parts fit together

Model loader

The runtime's input is a file, so the file format is the first contract. ONNX stores the model as a protobuf message (onnx.ModelProto) holding the graph, operator-set versions, and the weights as initializers (ONNX IR specification, see Sources). Decoding it is a walk over Protobuf wire format (no page yet) fields: tags, varints and length-delimited blobs, with weight tensors as large byte blobs. Protobuf messages have a 2 GB size limit, so large ONNX models keep their tensors in separate external files referenced from the model (ONNX External Data doc). Other formats pick differently: GGUF (used by llama.cpp) is a flat binary with a metadata header and a tensor table, aligned so the file can be mapped straight into memory (GGUF spec, "mmap compatibility"); ORT's own .ort format is a smaller serialised graph meant for reduced builds on mobile. Model file formats (no page yet) is the block that compares them; the runtime-level choice is whether the file is a graph description (portable, needs an optimizer) or a pre-lowered plan (fast to load, tied to a back end and version).

Weights dominate the size, so the loader should not copy them. With mmap and the page cache, weights become read-only file-backed pages: load time is near zero, the operating system shares one copy among processes, and pages are fetched on first touch. The cost is a slow first request (page faults) and tail latency if the file is evicted under memory pressure; for latency-critical serving the weights are touched once at startup (prefetched or "warmed") so the faults happen before traffic arrives.

Graph analysis

Shape inference (no page yet) runs operator by operator and fills in the shape of every tensor from the input shapes and the operator rules. Static shapes make everything downstream a constant; dynamic dimensions (batch size, sequence length) are carried symbolically or fixed per call. Dataflow analysis finds which nodes depend only on constants (to fold at load time), which tensors are read by several consumers (which blocks fusion), and when each tensor is last read (which the memory planner needs). The result is a graph annotated with shapes, types, constants and lifetimes.

Graph optimizer

ONNX Runtime divides its graph optimizations into levels (graph-optimizations doc, see Sources). Basic rewrites are semantics-preserving and run before partitioning, so they apply to every back end: constant folding, removal of redundant nodes (identity, dropout in inference), and simple fusions such as folding a bias add or batch norm into a convolution. Extended rewrites are complex fusions and run after partitioning, only on the nodes assigned to the back ends that implement them. Layout optimizations (for example NCHWc blocking for CPU convolution) run after partitioning and apply to the CPU back end. The order is the lesson: portable, back-end-independent rewrites first; rewrites that need to know the hardware only after nodes have been assigned.

Kernel fusion is the rewrite with the largest effect: a fused node reads its input once and writes its output once, instead of writing and re-reading every intermediate. Tensor layout transforms (no page yet) change the order of dimensions to what a kernel wants; each change that has to be undone later costs a full transpose, so the optimizer must count it. Optimized graphs can be saved (offline mode) so the next process skips the work; the docs note that the saved graph must be used with the same back ends and hardware it was optimized for, because the rewrites embed hardware-specific nodes.

Partitioner

Operator graph partitioning (no page yet) is what makes the runtime portable. In ONNX Runtime each execution provider implements GetCapability(), returning the nodes or subgraphs it can run (execution-providers doc). The application passes an ordered list of providers, for example CUDA first and CPU last; each node goes to the first provider that claims it, and the default CPU provider is the fallback for everything else. A back end that claims whole subgraphs (TensorRT, OpenVINO) is handed a connected region, which it compiles into one engine; a back end that claims single nodes (the CPU provider) is called node by node.

The contract: the partitioner owns the cut, the back end owns the kernels inside. The price is at every cut edge: a tensor leaving a GPU subgraph to a CPU node is a device-to-host copy. A graph that alternates between two back ends many times can be slower than one that stays on a single back end, even if each back end is faster per node. The practical rule is to prefer fewer, larger partitions and to keep an unsupported operator from splitting a hot region (rewrite it, or add the operator to the back end).

Memory planner

Intermediate tensors are allocated by a plan, not by malloc per node. Tensor memory planning (no page yet) uses the lifetimes from the analysis to assign each tensor an offset in a single buffer, reusing the bytes of tensors that are no longer alive. The buffer comes from an Arena allocator: one large block obtained up front, handed out by moving a pointer, freed in one step. Per request the runtime allocates nothing; peak memory is known after planning and does not drift.

The GPU back end uses the same idea with a pool (the CUDA provider documents gpu_mem_limit and an arena_extend_strategy option, see Sources). Two details matter in production: planning needs known shapes (with dynamic shapes the arena is sized for the largest request seen, or the plan is recomputed), and one session's arena is not shared between concurrent requests unless the session is run with separate buffers per call.

Executor

The executor needs an order in which every node runs after its inputs: a Topological sort of the graph, computed once and stored. ONNX Runtime has two modes: sequential (the default) runs nodes one at a time, and parallel mode runs independent branches of the graph concurrently, controlled by the intra-op and inter-op thread counts (thread-management doc). Parallelism inside an operator (splitting a matmul's rows among cores) is the intra-op pool; parallelism between operators is the inter-op pool. The intra-op default is one thread per physical core.

Both pools consist of many short tasks, which is the case for a Work-stealing deque: each worker owns a local queue and idle workers steal from others, so load balances without a shared lock. The docs also describe thread spinning (waiting threads busy-loop for a bounded time before sleeping), which lowers latency per call and costs CPU time; for a service with many sessions on shared cores, turning spinning off or limiting it is a common setting.

Kernel library

Each back end ships kernels. On CPU, Tiled matrix multiplication is the core of the dense operators: blocks sized to cache, an inner kernel written with SIMD programming (vector registers and fused multiply-add). On GPUs, kernels are written in a vendor language or as GPU compute shaders, which is also how a browser or cross-vendor back end (WebGPU) runs the same operators without CUDA. Weight quantization stores weights as int8 or int4 with scales: the file is smaller, memory traffic per token drops, and the kernel dequantizes on the fly or uses integer instructions. The contract with the planner is that quantized operators have their own dtypes and shapes, so they need a kernel per type, which is where operator coverage gaps usually come from.

GPU launch layer

On a fast GPU with small kernels, launching kernels from the CPU can cost more than running them. CUDA graphs (no page yet) record a sequence of kernel launches once and replay it with one call. ONNX Runtime's CUDA provider exposes this as the enable_cuda_graph option (documented as preview). Its documented constraints define the contract: every node must be assigned to the CUDA provider, control-flow operators are not supported, and shapes and addresses of inputs and outputs must not change between calls, so inputs are bound with I/O binding and copied into the fixed buffers. Several graphs can be captured per session, selected by a gpu_graph_id run option, which is how multiple input shapes are handled. It pairs with the memory plan: fixed offsets make a graph replayable.

Data and control flow

Loading a session (once).

1. Open the model file; parse it (protobuf decode, or map a flat file); map or load the weights.
2. Infer shapes and run the basic rewrites; fold constants.
3. Partition by the provider list; each provider compiles or prepares its subgraphs; run extended and layout rewrites.
4. Plan memory, create the arenas and thread pools, optionally record GPU graphs, and warm the weights.
5. Optionally save the optimized graph so the next load skips steps 2 to 4.

One request.

1. Bind the inputs (host tensors, or tensors already on the device to avoid a copy).
2. The executor runs the stored order. Each node: look up input pointers from the plan, call the kernel.
3. Intra-op parallel kernels split work across the pool; GPU kernels are queued on a stream; replay launches a whole recorded graph.
4. Copy the outputs back if they must leave the device.

Time goes where the model spends it: matmul-heavy nodes dominate, then memory-bound elementwise nodes (which fusion shrinks), then copies at partition edges and kernel launch gaps on GPUs.

Scaling

In the order you would pull the levers:

1. Per-request latency: optimization level on, offline-optimized model, fused kernels, weights warmed, a GPU graph for small kernels.
2. Throughput per machine: batch requests (dynamic batching in front of the runtime), tune the intra-op thread count to the cores per session, run several sessions with fewer threads each instead of one with all of them.
3. Memory: quantize the weights, share weights between sessions through mmap, size arenas to the real maximum shape.
4. Operator coverage: every unsupported operator is a partition cut; add the kernel, rewrite the model, or accept the fallback on a cold path.
5. Fleet: the runtime is stateless per request, so scale by adding replicas; the model file is the only artifact to distribute and it should be version-pinned together with the runtime and provider versions.

Failure modes

Real implementations

Sources

• ONNX Runtime docs: execution providers (GetCapability(), provider priority order), graph optimizations (basic, extended, layout; online and offline), thread management (intra-op and inter-op, spinning), ORT format models, CUDA execution provider (enable_cuda_graph, gpu_mem_limit).
• ONNX IR specification; ONNX External Data.
• GGUF specification.
• OpenVINO docs, TensorRT docs.