code-completion

How it is built

Architecture overview

Inline code completion shows grey "ghost text" at the cursor, a guess at what the programmer is about to type, and accepts it with one key. It is a request loop with a latency budget of a few hundred milliseconds that must run on a fraction of keystrokes, never block typing, and show a suggestion only when it is probably right. A small code language model, trained to fill in the middle of a file given the text before and after the cursor, does the guessing. Around it sit the parts that decide quality: a context builder that picks what the model sees (the code around the cursor and relevant snippets from other files), a request manager that drops stale work, a serving layer that reuses computation across requests, and a filter that hides low-confidence suggestions. The one idea that makes it work: the model is only a third of the product. Most of the quality comes from the prompt you assemble, and most of the latency comes from the work you avoid: requests never sent, prompts already cached, tokens never generated. Acceptance data from users closes the loop.

Parts:

• Editor client — trigger on typing, wait a moment, cancel when the user types on, show ghost text, handle Tab — debouncing-throttling (no page yet), Language server protocol, Incremental parsing
• Context builder — build the prompt from prefix, suffix and related code — fill-in-the-middle (no page yet), code-context-retrieval (no page yet), Code symbol indexing, Hybrid search rank fusion
• Model and decoding — a fill-in-the-middle code model, short and stoppable generation — Token sampling, token-healing (no page yet), Constrained decoding, Transformer block
• Serving — keep time to first token low under thousands of concurrent users — Prompt caching, Paged KV cache, Continuous batching, Speculative decoding, Weight quantization, Knowledge distillation
• Post-processing and cache — trim the completion, check syntax, reuse recent answers — Incremental parsing, LRU and CLOCK
• Measurement — accept rate, retention, experiments — A/B testing, Evaluation metrics

 keystroke ─► debounce ─► cancel older request ─► build prompt ─────────────► inference server
  (editor)     (wait)         (in flight)         prefix │ suffix │ snippets    ┌───────────────────┐
     ▲                                              ▲        ▲                 │ prefix cache hit? │
     │                                  open files, │        │ symbol index,   │ prefill ► decode  │
     │ ghost text                       recent edits│        │ syntax tree     │ (FIM model, short)│
     │                                              │        │                 └─────────┬─────────┘
 accept (Tab) ◄── show? ◄── filter, trim, syntax ◄─┴────────┴── completion ◄────────────┘
     │            (score)       check
     └──► log: shown, accepted, rejected ──► metrics, experiments, next model

How the parts fit together

Editor client

The editor talks to completion providers through its own API or, increasingly, through the Language server protocol: version 3.18 of the specification adds a textDocument/inlineCompletion request for exactly this ghost-text case, next to the older completion list that suits a dropdown of symbols. The LSP page covers the cost of keeping the server's copy of a live, edited document in sync; for inline completion the important contract is the request payload: document version, cursor position, trigger kind, and a way to cancel.

Three rules keep the client from hurting the user.

• Wait before asking (debouncing-throttling, no page yet). Typing produces a request per keystroke; sending them all wastes compute and returns suggestions for text that no longer exists. Continue's documentation, for example, shows a configuration with a debounce delay of 250 ms and a model timeout of 150 ms (an example configuration, not a universal setting).
• Cancel the previous request. When a new keystroke arrives, abort the in-flight call, and make the server stop generating, because tokens produced for an old cursor position are paid for and thrown away.
• Never render a stale answer. Every response is checked against the current document version and cursor; if the user has typed characters that match the start of the suggestion, the client trims it instead of discarding it.

The client also owns the syntax-aware parts that need to be instant: an incremental parser such as tree-sitter re-parses only the edited region on every keystroke (its README: fast enough to parse on every keystroke, robust in the presence of syntax errors), which tells the context builder whether the cursor is inside a function, a string or a comment, and tells the filter whether a completion leaves the file parseable.

Context builder

The model is only as good as its prompt, and the prompt has a fixed budget (Continue's example configuration caps it at 1024 tokens, with fractions of it reserved for prefix and suffix). Three sources go in:

• Prefix and suffix. Text before the cursor and after it, in the fill-in-the-middle format (fill-in-the-middle, no page yet). The format comes from training: a span from the middle of a document is moved to the end, so an ordinary left-to-right model learns to infill, and the study of this transformation by Bavarian et al. (2022) found it did not hurt left-to-right ability. The model is prompted with special sentinel tokens for prefix, suffix and "middle", then generates the middle. The suffix matters: without it the model repeats code that already follows the cursor or closes brackets that are already closed.
• Related code from the repository (code-context-retrieval, no page yet). Definitions of symbols used near the cursor, recently edited chunks, open files and similar snippets found by lexical or vector search, ranked and cut to size. Code symbol indexing provides the "where is this function defined" lookup; Hybrid search rank fusion merges keyword and embedding hits. llama.vim shows a minimal version of the idea: a ring of context chunks from open files, edited regions and yanked text, plus reuse of the server's cached context so that large contexts stay affordable.
• Metadata. File path, language, imports, and indentation of the current line.

The contract between this part and the model is a single string with sentinel tokens in the exact format the model was trained with. Continue's documentation is blunt about the consequence: strong general chat models that were not trained with the fill-in prompt format do not produce useful completions, and most state-of-the-art autocomplete models are no larger than about ten billion parameters.

Model and decoding

The model is a decoder-only Transformer trained on code with a large fraction of fill-in-the-middle examples; StarCoder and Code Llama are published open examples of code models with infilling. Decoding is deliberately different from chat:

• Short and stoppable. Completions are a line or a block; generation stops at a newline, a closing bracket of matching depth, a token budget, or a wall-clock timeout. Generating less is the cheapest speed-up there is.
• Low-randomness sampling. Greedy or a low temperature, since the user wants the most likely continuation, not a creative one.
• Token healing (token-healing, no page yet). A prompt that ends in the middle of a token, such as after http:, puts the model in a distribution it rarely saw in training, because tokenizers merge characters greedily; the Guidance project's notebook on prompt boundaries shows a generated URL breaking for that reason. The fix is to back up one token and constrain the first generated token to start with the removed text. Code ends mid-identifier all the time, so this matters here more than in chat.
• Optional constrained decoding to keep output inside a grammar, for instance balanced brackets, when a cheap structural guarantee is worth the mask computation.

Serving

An inline-completion fleet sees a high rate of small requests whose prompts mostly repeat the previous one. Cursor says its Tab model handles over 400 million requests per day, which is on the order of 4,600 a second on average (arithmetic from that figure). The serving layer exploits the repetition:

• Prompt caching keeps the attention keys and values of a shared prompt prefix, so the next keystroke's request prefills only the new tokens. The context builder cooperates by placing stable material (file header, retrieved definitions) before volatile material (the lines at the cursor) and by not reshuffling snippets between requests.
• Paged KV cache stores those caches in fixed blocks so thousands of live prefixes share GPU memory without fragmentation; the PagedAttention paper describes the idea and vLLM implements it.
• Continuous batching adds and removes requests each decoding step instead of waiting for a batch, which keeps a GPU busy without making a short request wait for a long one.
• Speculative decoding lets a cheap draft propose tokens that the main model verifies in one pass. Code edits are an ideal case, because much of an edit is copied from the existing file: Cursor describes a variant called speculative edits that reached about 1,000 tokens per second on a 70-billion-parameter model for whole-file rewrites.
• Weight quantization and knowledge distillation shrink the model so that time to first token fits the budget; a distilled small model that was trained on the fill-in task usually beats a quantized large chat model on this workload.

The contract with the context builder is stable-prefix ordering; the contract with the client is a streamed response that stops when cancelled.

Post-processing, filter and cache

The raw completion is not shown yet. The post-processor trims trailing text that duplicates the suffix, cuts the completion at a sensible boundary (end of statement, end of block), re-indents it, and checks that the document still parses with the incremental parser. Then a show or hide decision: many systems train a small classifier on acceptance data so that a low-probability suggestion stays hidden, because a wrong suggestion costs attention. Cursor's write-up describes GitHub Copilot's published contextual filter as a logistic-regression score over a handful of features (language, whether the last suggestion was accepted, the characters before the cursor and others) with suggestions below a threshold skipped; Cursor's own approach replaces the separate filter with reinforcement learning on acceptance, with a reward that shows a suggestion only when its estimated accept chance exceeds a target (their example is 25 %). A small LRU/CLOCK cache keyed by the exact prompt returns an immediate answer when the user deletes a character and retypes it.

Measurement

The product metric is the accept rate and what survives afterwards. Ziegler et al. found that the rate at which shown suggestions are accepted, rather than how long completions persist in the code, was what best matched developers' perceived productivity in the Copilot study. That makes acceptance both the thing to optimise and a dangerous target, since showing fewer, safer suggestions raises it; pair it with the number of suggestions shown per hour and per-edit retention. Changes ship behind an A/B test with the guardrails listed there. Offline evaluation metrics on held-out fill-in tasks screen models before they reach users.

Data and control flow

One keystroke to one suggestion:

1. The editor sends a change event; the incremental parser updates the syntax tree for the edited region.
2. A debounce timer restarts. Any in-flight request is cancelled.
3. After the delay, the context builder assembles prefix, suffix and snippets, ordered with stable parts first, within the token budget.
4. The server finds the longest cached prefix, prefills only the remainder, and decodes with a stop rule and a timeout; the first tokens stream back.
5. The post-processor trims, checks syntax and scores; the filter decides to show or hide.
6. The client renders ghost text; Tab accepts it; any other key dismisses or narrows it. Where time goes: debounce delay, time to first token (dominated by prefill of the uncached part), and the few tens of decoded tokens. Cutting any one wins more than a faster GPU.

Acceptance feedback and the next model:

1. Each shown suggestion is logged with its context hash, outcome (accepted, rejected, ignored) and latency.
2. Offline, outcomes become labels for the filter or a reward for training. Cursor reports rolling out new models to users frequently throughout the day and training on that on-policy data, so the model that is learning is the one that is being judged.
3. A candidate model is evaluated offline, then A/B tested on live traffic against the current model before being promoted.

Indexing the repository:

1. A background job parses files with the incremental parser and extracts definitions and references.
2. Only files whose content hash changed are re-indexed (Code symbol indexing); embeddings, if used, are stored in a vector index.
3. At request time the context builder queries by the symbols near the cursor and gets ranked snippets back in a few milliseconds.

Scaling

In the order you would pull the levers:

1. Send fewer requests. Debounce, cancel, skip positions where a suggestion is unlikely (inside a comment, after a closing line), serve repeats from cache. Everything saved here is saved at every layer below.
2. Reuse prompt computation. Stable-prefix ordering and prompt caching; prefill of the new tokens, not the whole prompt.
3. Shrink the model. A small, fill-in-trained or distilled model, quantized; accept lower raw capability for lower latency, since the context carries much of the quality.
4. Raise GPU utilisation. Continuous batching, paged KV cache, speculative decoding for edits.
5. Move computation closer. Regional inference near users, or a local model (llama.cpp, Ollama, Tabby) for privacy and zero network time, at the cost of a smaller model.
6. Improve quality per request. Better retrieval, more fill-in training data, learning from acceptance; these raise accept rate without extra latency.

Failure modes

Real implementations

The alternative: a chat or agent model asked to "complete this code" gives better reasoning for large changes but is too slow and too chatty for keystroke-level completion. The two coexist: completion handles the next few tokens or lines in the editor, an agent handles tasks that span files.

Sources

• M. Bavarian et al., Efficient Training of Language Models to Fill in the Middle, 2022. paper
• A. Ziegler et al., Productivity Assessment of Neural Code Completion, MAPS 2022. paper
• Language Server Protocol 3.18 specification (textDocument/inlineCompletion). spec
• J. Jackson, P. Kravtsov, S. Jain, Improving Cursor Tab with online RL, Cursor, 2025. post
• A. Sanger, Editing Files at 1000 Tokens per Second, Cursor, 2024. post
• Continue documentation, Autocomplete. docs
• Guidance, Prompt Boundaries and Token Healing. notebook
• R. Li et al., StarCoder: may the source be with you!, 2023. paper
• B. Rozière et al., Code Llama: Open Foundation Models for Code, 2023. paper
• W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. paper
• Y. Leviathan, M. Kalman, Y. Matias, Fast Inference from Transformers via Speculative Decoding, 2022. paper