model-serving

How it is built

Architecture overview

A model-serving system turns a trained model file into a network service with a latency and a cost you can plan for. It has a registry (the durable record of which model versions exist, where their files are, and which one is allowed to serve), a serving runtime (a process that loads a model onto an accelerator and runs requests through it in batches), a router in front of many runtime replicas, and a control plane that decides how many replicas run and when a new version replaces an old one. The one idea that makes it work is that the accelerator is efficient only when it is busy with a batch, so almost everything else, from the wire format to the autoscaler, exists to keep a batch ready without making any single caller wait too long. Triton Inference Server and KServe are the two best-known open-source halves of this: Triton is the runtime, KServe is the Kubernetes control plane that deploys runtimes such as Triton.

Parts:

• Model registry — versioned models, metadata, stage labels, file locations, lineage — Model registry (no page yet), Point-in-time recovery, Primary-backup replication
• Model preparation — turns a checkpoint into something fast to serve — Weight quantization, Kernel fusion, CUDA graphs (no page yet), LoRA adapters (no page yet)
• Serving runtime — loads models, queues requests, forms batches, runs them — Dynamic batching, Zero-copy serialization
• Wire protocol — how clients send tensors and get results — HTTP/2, Protobuf wire format (no page yet), Zero-copy serialization
• Router — spreads requests over replicas and protects against sick ones — Power of two choices, Circuit breaker, Mutual TLS
• Orchestration — places replicas on machines and scales them — Containers and Kubernetes, Autoscaling (no page yet)
• Rollout and experiments — moves traffic to a new version safely and measures it — Staged rollout (no page yet), A/B testing (online controlled experiments)
• Operations — keeping processes alive, deploys invisible, latency measured, and the whole thing tested — Process supervision, Graceful draining, HDR histogram (no page yet), Deterministic simulation testing, Coverage-guided fuzzing

 trainer ──register──► model registry ──version + stage──► controller (rollout, autoscaler)
 (checkpoint,          (DB + object store)                       │ desired replicas, traffic split
  quantized copy)                                                ▼
                       model store ──pull on start──► replicas 1..N, each:
                       (files per version)              queue ► batcher ► model on GPU
 client ──gRPC/HTTP──► router (P2C, breaker, mTLS) ──►  metrics: queue time, batch size, latency
        ◄───────────── result ◄─────────────────────────────┘
                                                              │
                                 metrics store ◄──────────────┘ ─► autoscaler (queue depth or concurrency)

There are two paths. The request path (router, runtime, accelerator) carries the latency budget. The management path (registry, controller, rollout) changes what the request path runs, and must never be required for an already-running replica to answer requests.

How the parts fit together

Model registry

The registry (Model registry, no page yet) answers "which bytes are version 7 of this model, who produced them, and is it allowed in production". Its usual shape is a metadata database plus an object store for the files. A version carries its files, the metrics measured on it, a link to the training run and data, and a stage or alias (candidate, production). The serving side asks the registry for "the production version" or for a pinned version, never for a path. The contracts that matter:

• Immutability. A registered version never changes; a fix is a new version. Otherwise a rollback is not a rollback.
• One source of truth for the alias. The controller reads "production" from the registry; moving the alias is the act of promotion and is logged.
• Durability. Losing the metadata database loses the mapping from names to files, so it gets Point-in-time recovery and Primary-backup replication.

MLflow is the best-known open-source registry (see Real implementations). Triton takes a simpler approach: the model repository is a directory tree, one folder per model with numbered version subfolders and a config.pbtxt, which can be a local path or object storage.

Model preparation

Preparation is where a checkpoint becomes a deployable artifact, and it is mostly about moving fewer bytes and launching fewer kernels. Weight quantization shrinks weights to 8 or 4 bits, which cuts memory (more model per GPU, more room for batches) and, for decoder models limited by memory bandwidth, speeds up each step. Kernel fusion merges chains of small operations into one kernel so intermediates stay in fast memory; compilers such as TensorRT do this when they build an engine from a graph. CUDA graphs (no page yet) record a sequence of GPU launches once and replay it, removing CPU launch overhead for small, repeated shapes. LoRA adapters (no page yet) let many fine-tuned variants share one base model: the runtime keeps the base weights resident and applies a small per-tenant delta per request, instead of loading a full copy of the model for each tenant.

The contract with the registry: the prepared artifact is registered as its own version with its provenance (source version, quantization settings, calibration data) and its measured accuracy loss, so a promotion gate can compare it with the original.

Serving runtime

The runtime owns the accelerator. A request arrives, waits in a per-model queue, is grouped by the Dynamic batching policy (dispatch when the batch reaches a preferred size or the oldest request has waited a configured maximum delay), run once, and split back into individual responses. In Triton the knobs are in the model's config.pbtxt: max_batch_size, instance_group (how many copies of the model per device), and the dynamic_batching block with preferred_batch_size and max_queue_delay_microseconds (model configuration, batcher). The queue delay is the latency you deliberately trade for throughput, so it is set from the latency budget and not from the benchmark's best throughput.

Large language models change one thing: a request lives for hundreds of steps, so batching is done per step (continuous batching) and the key resource is the attention cache of each running sequence. vLLM manages that cache in fixed-size blocks, an approach called PagedAttention (Kwon et al. 2023, see Sources). The registry, router and rollout parts of this page are unchanged; the runtime and the autoscaling signal (below) differ.

Triton can also chain models inside one server (ensembles, plus Python business-logic scripting), so pre-processing, the model and post-processing run without a network hop (ensemble models).

Models are loaded and unloaded by a control mode: Triton's --model-control-mode is none (load at start), poll (watch the repository) or explicit (load on API call) (model management). Explicit control is what a controller wants, because it chooses exactly when a version is loaded and warmed.

Wire protocol

Inference requests are tensors: a shape, a datatype and a block of bytes. Two framings dominate. gRPC runs over HTTP/2 with messages in Protobuf wire format (no page yet): one connection carries many concurrent requests (multiplexing), and the schema is typed. JSON over HTTP is easier to debug, but encoding a large float tensor as text costs CPU and bytes, so serving protocols add a binary tensor extension. The KServe v2 protocol, now maintained as the Open Inference Protocol, defines the same request shape for HTTP and gRPC so that clients and runtimes are interchangeable (open-inference-protocol). Inside the server, Zero-copy serialization matters: the bytes that arrived on the socket should reach the GPU copy without an intermediate decode into another buffer.

Router

In front of the replicas, a router (an ingress, a service mesh sidecar, or the client library) sends each request to one replica. Power of two choices (pick two replicas at random, send to the one with less queued work) gives near-ideal balance without a global view, and it adapts when a replica slows down. Round-robin ignores the fact that replicas hold different queue lengths. Affinity is the exception: sticky routing of a tenant's LoRA adapter to a replica that already has it loaded, with spill-over to P2C when that replica is full.

A Circuit breaker per replica stops sending to one that times out or errors, so a stuck GPU costs a few requests and not a timeout for every Nth request. Traffic between router and replicas should use Mutual TLS when replicas are shared across teams, since model inputs are often the most sensitive data in the company; a service mesh is the usual way to get it without changing the runtime.

Orchestration

Containers and Kubernetes place replicas: each replica is a container with the runtime, a GPU resource request, and a readiness probe that turns true only after the model is loaded and warmed. The pull of a multi-gigabyte model file at start is the main cost of a new replica, so replicas cache the model files on node disk or a shared volume.

Autoscaling (no page yet) chooses the replica count. CPU utilisation is a poor signal for GPU services; the signals that work are concurrent requests per replica or queue time, because they rise before latency does. KServe builds on Knative for request-based scaling, including scale to zero for rarely used models; scale to zero trades cost for a cold start of the whole model load, so latency-critical models keep a minimum replica count (KServe). Scaling up is slower than request bursts, so pair it with a queue limit that sheds load instead of letting the queue grow.

Rollout and experiments

Replacing a model is a release like any other, with the extra risk that a new model can be worse without being broken: it returns well-formed answers that are less accurate. Staged rollout (no page yet) moves a small share of traffic to the new version, checks error rate, latency and a quality signal, and widens in steps; the old version stays loaded for instant rollback. KServe expresses the share as a canary traffic percentage on an InferenceService (canary rollout). Latency and errors can be judged in minutes; quality needs A/B testing with enough traffic and a metric that reflects the product, such as click-through, acceptance rate or a labelled sample.

Operations

A runtime that holds a GPU can crash on a bad input or a driver error. Process supervision restarts it, and Kubernetes liveness probes play the same role at pod level. Graceful draining makes a version swap or scale-down invisible: the replica leaves the router's pool, finishes queued and in-flight requests, and then exits; for long LLM generations the drain deadline must exceed the longest generation. Latency is recorded per model and per version in an HDR histogram (no page yet), because average latency hides exactly the tail that batching creates. Triton exposes Prometheus metrics (counts, queue and compute durations) on its metrics port, 8002 by default (metrics).

The batcher, queue and loader are concurrent code with timers, which is where Deterministic simulation testing pays off: simulate arrival bursts, slow batches and a replica dying mid-batch, then replay any failing seed. Request decoders (tensor shapes and sizes from the network) are targets for Coverage-guided fuzzing; a request that declares a huge shape must be rejected before allocation.

Data and control flow

One inference request:

1. The client sends a tensor request over gRPC or HTTP/2 to the router, which opens or reuses a connection.
2. The router picks a replica by P2C among those whose breaker is closed.
3. The runtime decodes the request without copying more than needed and appends it to the model's queue.
4. The batcher dispatches when the batch is full or the oldest request has waited the maximum delay.
5. The batch runs on the GPU (fused kernels, possibly a replayed CUDA graph); outputs are split per request.
6. The response goes back along the same connection. Time goes to queueing (a chosen delay), the model compute (usually the largest part), and the wire; the metrics record each separately.

One new model version:

1. The trainer registers the version with its metrics; a prepared (quantized, compiled) variant is registered as a sibling with its measured accuracy loss.
2. A gate compares it with production on a held-out set; passing moves it to candidate.
3. The controller starts replicas of the candidate and waits for readiness (loaded, warmed).
4. Traffic moves in steps, for example a few percent, then more, watching error rate, latency and the experiment metric; failing a check sends traffic back and unloads the candidate.
5. On success the registry alias moves, the old version drains and is unloaded after a grace period.

Scaling

In the order you usually hit them:

1. Utilisation of one GPU: batch more (raise the max queue delay within the latency budget), run several model instances per device, fuse and quantize. This is the cheapest lever.
2. Latency tail: cap queue length and shed load, add replicas before queues grow, keep the tail in a histogram.
3. Replica count: autoscale on concurrency or queue time; keep a minimum for models where a cold start would break the latency objective.
4. Many models: thousands of rarely used models cannot each hold a GPU. Share GPUs between models, swap models in and out on demand (accepting a cold-start penalty), and share one base model across many LoRA adapters.
5. Model size: when one model no longer fits one GPU, shard it across GPUs (tensor or pipeline parallelism in the runtime); this raises the unit of scaling from one GPU to a group.

Failure modes

Real implementations

The alternative: a managed endpoint from a cloud provider or an API company hides the registry, runtime and autoscaler behind one call. That removes the operations work and fixes the model, the batching policy and the price; build the pieces above when you need your own models, your own hardware, or tight control of latency and data location.

Sources

• W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. paper
• G.-I. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022 (iteration-level scheduling, the origin of continuous batching). page
• NVIDIA Triton user guide: model repository, model configuration, dynamic batcher.
• KServe documentation: serving runtimes, inference graph.
• MLflow model registry documentation, linked in the table.