llm-api-platform

How it is built

Architecture overview

An LLM API platform sells access to a fixed set of large models over HTTP. A request carries a prompt, sometimes tools, and gets back tokens, usually streamed. Behind that one call sit four very different systems: an edge that authenticates, limits and meters traffic; a router that picks a model and a replica pool; an inference fleet of GPU servers whose cost per token decides the business; and a control plane that decides which models run where, in which tier, with which rollout state. The central difficulty is that GPU capacity is scarce, slow to add, and spent in very uneven units: a request may use 200 tokens or 200,000, and its cost depends on how much of the prompt is already in a cache.

The idea that makes it work is that every layer shares one currency, the token, and one scarce resource, KV-cache memory on a GPU. Rate limits count tokens, billing counts tokens, schedulers pack tokens into batches, caches reuse the key-value state of tokens already computed. Priority, batch and cached traffic are then three ways to sell the same GPU seconds at different prices and different promises. Public documentation of the big providers shows the visible edge of this: token-bucket limits and streamed events at the API (source: Anthropic, Rate limits; Streaming Messages), 5-minute and 1-hour prompt-cache lifetimes, and a batch API at a 50 % discount (source: Anthropic, Prompt caching; Batch processing). The inside is mostly assembled from open serving engines and papers.

Parts:

• Edge and admission — authentication, per-key and per-tier limits, retries without double charges, overload protection — Distributed rate limiting, Adaptive concurrency limits (no page yet), Idempotency keys, Mutual TLS, Circuit breaker
• API surface — streaming responses, tool calls, structured output, server-side tools — Server-sent events, Tool calling, Constrained decoding, Model Context Protocol (no page yet)
• Router — picks model version, region and replica pool for each request — Model routing and cascades, Prompt caching, Staged rollout (no page yet)
• Inference engine — turns a batch of requests into tokens on a group of GPUs — Continuous batching, Paged KV cache, Tensor parallelism, Speculative decoding, Prefill/decode disaggregation (no page yet)
• Capacity and scheduling — decides who gets GPU time and how much hardware is warm — Autoscaling (no page yet), Warm-pool provisioning (no page yet), Priority and preemption (no page yet), Fair-share scheduling (no page yet)
• Metering — counts input, output and cached tokens per customer, exactly once — Usage metering pipeline, Idempotency keys
• Safety hooks — input and output classifiers, tool-use guard rails — Prompt injection defense, Constrained decoding
• Observability and operations — per-request traces, latency percentiles, safe restarts, upgrades — Distributed tracing, HDR histogram (no page yet), Graceful draining, Process supervision
• Control-plane state — customers, keys, limits, model registry, rollout state — Primary-backup replication, Point-in-time recovery
• Verification — finds the bugs that only appear under real failure — Deterministic simulation testing, Coverage-guided fuzzing

 client ──TLS──► edge: auth ─► rate/concurrency limits ─► idempotency ─► safety pre-check
                                 │ 429 + retry-after            │ replay stored result
                                 ▼                              ▼
                       router: model, tier, region, prefix-hash ──► cache-affine replica
                                 │                                        │
          ┌──────────────────────┴──────────┐              ┌──────────────┴──────────────┐
          │ prefill pool (compute-bound)    │──KV blocks──►│ decode pool (memory-bound)  │
          │ continuous batch, paged KV,     │              │ continuous batch, spec.     │
          │ prefix cache                    │              │ decoding, TP across GPUs    │
          └─────────────────────────────────┘              └──────────────┬──────────────┘
                                                                          │ tokens (SSE)
 metering log ◄── token counts, cache hits ──── edge ◄── safety post-check ┘
 control plane: model registry, rollout state, limits ──► router, autoscaler, warm pool

How the parts fit together

Edge and admission

The edge must say no cheaply: a rejected request costs microseconds, an accepted one GPU seconds. Three limits stack. A rate limit over tokens per minute, enforced across many edge servers by Distributed rate limiting: the usual choice is a local bucket per node with batched leases from a shared counter, because a central counter per request would add a network hop to every call. Anthropic documents its limits as a token bucket that refills continuously, so capacity is replenished up to a maximum rather than reset on the minute (source: Anthropic, Rate limits). A concurrency limit on in-flight requests, adjusted by Adaptive concurrency limits (no page yet): it watches latency and shrinks the limit when the fleet slows down. A spend or tier limit from the control plane. Anthropic also documents "acceleration" 429s for sharp usage increases, the sign of a limiter that looks at slope, not only level (source: Anthropic, Rate limits).

Clients retry, and a retry of a request that actually completed would charge twice and could run a tool twice. Idempotency keys give the edge a stored result to replay. The cost is a store lookup per request and a retention window; a reused key with a different body must be rejected, not served.

Inside the platform, Mutual TLS gives every hop a verified identity, and a Circuit breaker on each upstream stops a sick pool from piling up waiting requests.

API surface

A reply takes seconds, so the API streams it: Server-sent events carry text deltas over one HTTP response, with periodic keep-alive events so proxies do not close an idle stream. In Anthropic's documented stream, a message is a sequence of message_start, content blocks made of content_block_start, one or more content_block_delta and content_block_stop, then message_delta and message_stop, with ping events interleaved and token counts that are cumulative (source: Anthropic, Streaming Messages). An error can arrive mid-stream, after the 200 status was sent, so clients must treat the stream as the result.

Tool calling is the contract that lets the model request an action: the model emits a structured call with an id, the client or a server-side tool runs it, and the result returns under the same id. Where tools run on the provider's side, the platform becomes an executor and needs sandboxing, per-tool timeouts and Prompt injection defense, because a fetched web page is untrusted text that lands in the model's context. Constrained decoding enforces a JSON schema by masking tokens at each step; it runs inside the engine, but schema compilation is an API-level feature with its own limits. The Model Context Protocol (no page yet) is the open interface through which a platform reaches customer-hosted tools.

Router

The router answers: which model version, which region, which replica. Model choice by the customer is explicit, but behind a model name sit several versions at different rollout stages (Staged rollout, no page yet), and a platform may also offer cheaper cascades (Model routing and cascades). Replica choice is where the money is: Prompt caching only pays if a request lands on a replica that still holds the KV blocks of its prefix. So the router hashes the prompt prefix and prefers the replica (or pool) that served it last, falling back to the least loaded replica when the preferred one is saturated. Documented behaviour at the API matches this: caches have a 5-minute default lifetime that is refreshed on use, an optional 1-hour lifetime, and a lookback of 20 blocks per breakpoint (source: Anthropic, Prompt caching). A cache is a routing property, not a global one.

Inference engine

One replica is a group of GPUs holding one model copy. Tensor parallelism splits each layer across the group (Megatron-LM style: column then row splits with one all-reduce per block, Shoeybi et al. 2019) so a model larger than one GPU's memory runs at the latency of one layer's math. The engine's scheduler uses Continuous batching: at every decoding step, finished sequences leave and waiting ones join, so the batch stays full whatever the request lengths (iteration-level scheduling, Yu et al., Orca, OSDI 2022). The memory for per-sequence attention state is managed as fixed blocks by a Paged KV cache, which removes fragmentation and lets requests that share a prefix share blocks (Kwon et al., PagedAttention, SOSP 2023). Block sharing makes the prompt cache cheap.

Speculative decoding lowers latency per token: a small draft model proposes several tokens and the large model verifies them in one pass, with rejection sampling keeping the output distribution exact (Leviathan et al. 2023). It trades extra compute for latency, so it pays where batches are small.

Prefill (reading the prompt) is compute-bound; decode (generating) is memory-bandwidth-bound. Running both in one batch makes long prompts stall everyone's next token. Prefill/decode disaggregation (no page yet) puts them in separate pools and ships the KV cache between them, so each pool is sized and tuned for its own latency target (DistServe, Zhong et al. 2024; Splitwise, Patel et al. 2024). The contract between the pools is the KV block layout and a transfer fast enough not to cancel the gain.

Capacity and scheduling

GPUs take minutes to start (load weights, compile, warm caches), so Autoscaling (no page yet) needs a signal that leads demand and a Warm-pool provisioning (no page yet) of loaded replicas to absorb a spike. The signal is queue depth or time-to-first-token, not CPU.

Traffic classes are the way to sell spare capacity. Interactive requests have a latency target; batch requests accept delayed completion for a lower price (Anthropic documents a 50 % discount; source: Anthropic, Batch processing); both run on the same fleet. Priority and preemption (no page yet) lets interactive work evict batch work, whose KV blocks are dropped or swapped out and recomputed later. Fair-share scheduling (no page yet) stops one large customer from taking every slot inside a class.

Metering

Billing is exact where limits are approximate. The engine reports input, output, cache-write and cache-read tokens per request; the edge writes one usage event per request with the request id as dedup key, and the Usage metering pipeline aggregates them per customer and period. A retried request must not bill twice, which is why metering reuses the idempotency key and the request id. Cache reads are priced differently from fresh input, so the cached-token split belongs in the event.

Safety hooks

Safety checks sit at three points: classifiers on the incoming prompt, constraints during decoding, and checks on the output stream. They are on the latency path, so they run as small models beside the main one, and an output check on a streamed reply can only cut the stream, not unsend what was shown. Provenance rules from Prompt injection defense keep model-chosen arguments from reaching privileged tools.

Observability and operations

A request crosses edge, router, prefill and decode pools; Distributed tracing joins those spans so that a slow time-to-first-token can be pinned on queueing, prefill or transfer. Latency is recorded in an HDR histogram (no page yet), because percentiles matter more than means and must be merged across servers. Graceful draining is mandatory for streams: on shutdown a replica stops taking new requests and finishes generating, since killing it truncates paying customers' replies. Process supervision restarts crashed engines with backoff.

Control-plane state and verification

Customers, keys, limits and model registry are small, strongly consistent data. A Primary-backup replication of a relational store with Point-in-time recovery is enough; the data plane never waits on it, because each edge caches the entries it needs. The harder part is testing rare failure paths: Deterministic simulation testing for the stateful pieces (limiter leases, idempotency store, metering), and Coverage-guided fuzzing for the parsers of untrusted input (request bodies, schemas, tool results).

Data and control flow

One streamed request:

1. The client opens TLS and sends the request with an idempotency key. The edge authenticates, charges the token estimate against the rate-limit bucket, checks the concurrency limit, and looks up the idempotency key.
2. The router resolves the model name to a version and a pool, hashes the prompt prefix and picks a replica that probably holds the prefix.
3. The engine matches the prefix against its cached blocks, then runs prefill only on the uncached tail. Time here grows with uncached prompt length.
4. Decode runs inside the continuous batch: one token per step per sequence, with speculative drafts if enabled. Each token goes back through the safety check to the SSE stream.
5. On completion, the engine reports token counts (input, output, cache read/write); the edge appends one usage event and stores the final response against the idempotency key. Most time and cost are in steps 3 and 4: time-to-first-token is queueing plus prefill, inter-token latency follows the decode batch size.

One capacity change:

1. The autoscaler sees queue depth or time-to-first-token above target for a window.
2. It claims replicas from the warm pool, or starts new ones if the pool is empty (weight loading takes minutes).
3. The router adds a replica only after a health probe passes; its cache is empty, so it gets traffic gradually.
4. On scale-in, the replica is drained, not killed.

Scaling

Pulled in this order:

1. Cache hit rate. The cheapest token is one that is not recomputed. Levers: route by prefix, longer cache lifetime, advice to customers to put stable content first.
2. Batch size and memory. Bigger batches raise tokens per GPU second until KV memory runs out. Levers: paged KV cache, quantized KV, shorter maximum context in cheap tiers.
3. Latency of the two phases. Levers: speculative decoding, prefill/decode disaggregation when long prompts hurt tail latency, tensor parallelism.
4. More replicas. Embarrassingly parallel but slow to add: warm pools, then capacity reserved ahead of launches.
5. Class mix. Batch and low-priority work fills troughs; preemption keeps interactive latency intact.
6. The edge and control plane scale last: limiter state shards by key, the metering log by customer.

Failure modes

Real implementations

No provider publishes its whole platform; the table lists the open parts that implement the pieces above.

The alternative: one model for one customer needs none of the router, tiers or edge: run vLLM or SGLang behind an ordinary load balancer.

Sources

• W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. paper
• G.-I. Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022. paper
• Y. Leviathan, M. Kalman, Y. Matias, Fast Inference from Transformers via Speculative Decoding, 2023. paper
• M. Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019. paper
• Y. Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, OSDI 2024. paper
• P. Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting, ISCA 2024. paper
• R. Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, 2024. paper
• Anthropic API documentation: Rate limits, Streaming Messages, Prompt caching, Batch processing.
• OpenAI, Batch guide; Google, Gemini API context caching.
• Stripe, Designing robust and predictable APIs with idempotency. blog