embedding-service

How it is built

Architecture overview

An embedding service turns text (or images, or code) into fixed-length vectors such that similar meaning means nearby vectors, and serves that conversion to many callers at a steady price per token. Three jobs share the name: an online path (a user's query is embedded in a few milliseconds and searched against an index), a bulk path (a whole corpus is embedded once, then re-embedded whenever the model changes) and a search index that holds the results. The model is an encoder Transformer block stack trained with Contrastive learning; the service around it is mostly about not wasting compute on padding and repeats: batching by token count, packing short texts together, caching repeated inputs, and keeping the output small. The idea that makes it work is that an embedding is only meaningful relative to the model that produced it, so the model version is part of every vector's identity and of every cache key.

Parts:

• Model — encoder that maps token ids to one vector per input — Transformer block, embedding-inference (no page yet), Contrastive learning, Matryoshka embeddings (no page yet)
• Tokenizer — text to token ids, truncation, length accounting and billing unit — Byte-pair encoding (BPE) tokenization
• Batcher and packer — groups requests by token budget, removes padding — Dynamic batching, Sequence packing (no page yet)
• Inference kernels — attention and matrix multiplies on the accelerator or the CPU — FlashAttention, Weight quantization, SIMD programming
• Embedding cache — skips the model for texts already seen — LRU and CLOCK
• Vector index — stores vectors and answers nearest-neighbour queries — HNSW, IVF-PQ
• Bulk pipeline — embeds a corpus offline and re-embeds it for a new model — (uses the batcher and index above)

 online:  query text ─► cache? ──hit──► vector ─► index search ─► ids + scores
                          │ miss
                          ▼
                   tokenizer ─► batcher (token budget, length-sorted) ─► model (GPU/CPU) ─► pool + normalise
                                                                              │                  │
                                                                      cache write ◄──────────────┘
 bulk:    corpus shards ─► workers (tokenize, pack, embed in large batches) ─► vectors + model-version tag
                                                          └──► index build (HNSW graph or IVF-PQ codes)
 swap:    new model ─► embed everything ─► build second index ─► switch alias ─► drop old index

The online and bulk paths run the same model code but want opposite things: online wants the lowest latency at small batches, bulk wants the highest throughput at the largest batch the memory allows. Many deployments run them as separate pools sharing the model artifact.

How the parts fit together

Model

Most production text embedders are encoder-only transformers (BERT-like), though decoder-based embedders exist. The encoder reads the whole input with bidirectional attention (Transformer block) and emits one hidden vector per token (embedding-inference (no page yet) is the block for running a pretrained encoder at serving time). A pooling step collapses the per-token vectors to one: the first token's vector or the mean over tokens, depending on how the model was trained. The result is usually L2-normalised, so cosine similarity becomes a dot product, which is what the index computes.

The model gets its geometry from Contrastive learning: pairs known to be related (a question and its answer passage, a title and its body) are pulled together and other items in the batch are pushed away. Sentence-BERT is the standard reference for fine-tuning an encoder this way (Reimers and Gurevych 2019). Two properties matter for the service. First, the training task fixes what "similar" means, so a model trained for retrieval is not automatically good at clustering. Second, vectors from different models are not comparable: the service must tag every stored vector with the model id and revision.

Matryoshka embeddings (no page yet) train the first k dimensions of the vector to be a usable embedding on their own (Kusupati et al. 2022). The service can then store or return a shortened vector by truncating and re-normalising. OpenAI's embedding API exposes this through a dimensions parameter, with default sizes of 1536 for text-embedding-3-small and 3072 for text-embedding-3-large (guide). Shorter vectors shrink the index and speed up search at some loss of accuracy; the loss is measured on your own retrieval set, not assumed.

Tokenizer

The tokenizer (Byte-pair encoding (BPE) tokenization) is part of the model, not a utility: a different vocabulary gives different ids and different vectors, so it ships in the same artifact and is versioned with it. (Some encoders use WordPiece, a close relative; the contract is the same.) It also fixes the unit of cost: services meter and rate-limit by tokens. The model has a maximum input length (the OpenAI guide lists 8192 tokens for its three current-generation models); the tokenizer truncates or rejects longer input, and the policy is a visible API decision, since silent truncation hides the tail of a long document from search. Tokenization is CPU work and, at high request rates, a real part of the latency; run it in a separate thread pool from the accelerator loop.

Batcher and packer

The accelerator is efficient on large batches, and requests arrive one at a time. Dynamic batching collects requests until a budget is reached or the oldest has waited too long. For embeddings the right budget is tokens per batch, not requests per batch, because cost grows with total tokens and attention cost grows with length: a batch of 8 requests of 512 tokens is far heavier than 8 requests of 16 tokens. Hugging Face's Text Embeddings Inference advertises exactly this, "token-based dynamic batching" (docs).

Padding is the second waste. A batch is a rectangle, so a 10-token query batched with a 500-token document is padded to 500. Two fixes: sort or bucket requests by length before forming batches, and Sequence packing (no page yet), which concatenates the sequences into one long row and tells attention where each one ends, so no compute goes to padding. Packing needs the attention kernel to respect sequence boundaries; that is what the variable-length interface of FlashAttention provides. Bulk pipelines pack aggressively because they have time to sort a whole shard by length.

Inference kernels

FlashAttention computes exact attention without materialising the full length-by-length score matrix, so memory grows linearly with length and long inputs (thousands of tokens) become affordable. Embedding requests are mostly prefill-style (the whole input in one pass, no token-by-token generation), so the kernel work is dominated by large matrix multiplies and attention, which suits the accelerator well; there is no growing attention cache as in text generation.

Weight quantization (fp16 or bf16 as the baseline, int8 or lower when measured acceptable) raises throughput and cuts memory so more model instances fit per device. The check that matters for embeddings is not perplexity but retrieval quality (recall at k on a labelled set) and vector drift: how far the quantized model's vectors move from the reference model's. If drift is large, old and new vectors in the same index will disagree.

For CPU serving (small models, cost-sensitive or edge cases), SIMD programming is the mechanism: vectorised dot products, int8 dot-product instructions and cache-friendly matrix layouts. It also matters on the index side, where distance computations over many vectors are the inner loop.

Embedding cache

Real traffic repeats: the same popular query, the same document re-submitted unchanged. A cache keyed by (model id and revision, truncation and dimension settings, hash of the text) returns the stored vector with no model call. An eviction policy such as LRU and CLOCK keeps the hot set; CLOCK is cheaper on a concurrent cache because a hit sets a bit instead of reordering a list. The key must include everything that changes the output, or a model upgrade silently serves old vectors. Query caches pay off online; for bulk corpora, de-duplication of identical chunks before embedding gives the same saving.

Vector index

The index answers "which stored vectors are closest to this one". Exact search compares against all vectors, which is fine for thousands and hopeless for hundreds of millions. Two approximate structures are the standard choices:

• HNSW: a layered proximity graph; query time is low and recall high, memory is the full vectors plus the graph links, and inserts are supported incrementally. Choose it when the index fits in RAM and updates are continuous.
• IVF-PQ: vectors are assigned to coarse cells (inverted file) and stored as short product-quantization codes; search probes a few cells and compares compressed codes. Memory per vector drops by an order of magnitude or more, at lower recall. Choose it when the corpus is too large for RAM at full precision, often with a re-ranking step that fetches full vectors for the top candidates.

The contract between model and index: dimension, metric (cosine or dot product), normalisation, and model revision must match, and the index records them. Matryoshka truncation is the cheap lever between the two: the same model can fill a short-vector index for recall-oriented first-stage search and a full-length one for re-ranking.

Bulk pipeline

Embedding a corpus is a batch job: read shards, tokenize, length-sort and pack, run large batches at high accelerator utilisation, write vectors with their ids and model tag, build the index. It must be resumable (a failed shard re-runs alone) and idempotent (re-embedding a shard overwrites its vectors, no duplicates). When the model changes, the entire corpus is re-embedded into a new index while the old one keeps serving; the alias is switched when the new index is complete and validated. There is no safe way to mix old and new vectors in one index.

Data and control flow

One online query:

1. The API authenticates the caller and checks the token rate limit (tokens counted by the tokenizer).
2. The cache is consulted with the full key; a hit returns at once.
3. On a miss the text is tokenized, truncated or rejected at the model limit, and queued.
4. The batcher forms a batch by token budget (queries are short, so many fit); the model runs; pooled vectors are normalised and optionally truncated to the requested dimension.
5. Vectors are written to the cache and returned. If the caller asked for a search, the vector goes to the index (HNSW graph walk, or IVF cell probe plus code scan), and ids and scores come back.
6. Time goes mostly to the model forward pass for a cache miss and to index search for a hit; the queue wait is the knob that trades one against the other.

One model upgrade:

1. The new model is evaluated on a labelled retrieval set against the current one.
2. The bulk pipeline embeds all corpus shards with the new model, tagging each vector with the new revision.
3. A second index is built and checked for recall against the first on held-out queries.
4. Query traffic is moved to the new model and index together, never one without the other, usually via a single alias that names both.
5. The old index and cache entries are dropped after a rollback window.

Scaling

1. Padding and idle accelerators: length-bucketing, packing and token-budget batching; these double or triple effective throughput on mixed-length traffic before any hardware is added (order of, depends on the length mix).
2. Repeats: a cache for queries, de-duplication for corpora.
3. Model cost: a smaller or quantized model, lower precision, shorter maximum input length; verify on retrieval metrics each step.
4. Index memory: shorter vectors (Matryoshka), then IVF-PQ compression with re-ranking, then sharding the index across machines by cell or by id range with a merge of the top-k from each shard.
5. Query rate on the index: replicate the index (read replicas), since search is read-only between builds.
6. Bulk time: add workers; shards are independent, so throughput scales almost linearly.

Failure modes

Real implementations

The alternative: call a hosted embedding API. It removes the model, batcher and capacity planning, and costs a network round trip per request, a per-token price, and a dependency on the vendor's model revisions. Self-host when volume makes the per-token price larger than the cost of a GPU you can keep busy, when data cannot leave your network, or when you need a model fine-tuned on your own pairs.

Sources

• N. Reimers, I. Gurevych, Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, EMNLP 2019. paper
• A. Kusupati et al., Matryoshka Representation Learning, NeurIPS 2022. paper
• T. Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022. paper
• OpenAI embeddings guide (dimensions parameter, default sizes, maximum input) and Hugging Face Text Embeddings Inference documentation, linked above.