recommender

How it is built

Architecture overview

A recommender answers one question per request: out of millions of items (videos, posts, products), which few should this person see now, in what order? Scoring every item with a big model is impossible inside a request, so the system is a funnel. Cheap candidate generators cut millions to a few thousand; a ranker with rich features scores those; a re-ranker arranges the winners for variety and for learning; the page is served. YouTube's paper describes exactly this split as the classic two-stage dichotomy of a deep candidate generation model and a separate deep ranking model (source: Covington et al., Google Research page), and X's open-sourced home timeline follows the same layering with candidate sources, a light ranker, a heavy ranker, and mixing and filtering (source: twitter/the-algorithm README). The idea that makes it work is that every stage speaks in vectors and probabilities: users and items live in one embedding space that approximate-nearest-neighbour search can query in milliseconds, and the ranker's outputs are calibrated probabilities that can be combined into one value per item. Everything else (streams, a feature store, experiments) exists to keep what the models see in training identical to what they see when serving, and to prove that a change helps.

Parts:

• Event stream and aggregates — turns raw impressions, plays and likes into per-user and per-item counters — Windowed stream aggregation, Count-min sketch
• Feature store — one definition of each feature for training and serving — Feature store
• Candidate generation — shortlist from the whole corpus — Two-tower retrieval, Contrastive learning, Negative sampling, Matrix factorization for collaborative filtering, Embeddings, Transformer block
• Vector index — nearest-neighbour search over item vectors — HNSW, IVF-PQ, Top-k selection
• Ranker — probability of each outcome per candidate — feature-interaction-models (no page yet), Click-probability prediction (CTR models), Deep ranking models for recommendation, Feature hashing (the hashing trick), embedding-table-sharding (no page yet), Calibration
• Re-ranker and exploration — diversity, freshness, and learning about new items — maximal-marginal-relevance (no page yet), Multi-armed bandits
• Evaluation — offline metrics and online tests — Evaluation metrics, A/B testing (online controlled experiments)

 client ─► feed service ─► candidate sources ────────────────► ranker ─► re-rank ─► page
 (user id,  (context)      ├ two-tower ► ANN index (HNSW/IVF-PQ)  │   (calibrated   (MMR, bandit
  context)                 ├ follow graph / in-network            │    p(click),     slot, rules)
                           ├ item-item / popular                  │    p(like) …)        │
                           └ fresh / exploration                  │                      │
                                  ▲ ~10^3 each                    ▲ ~10^2-10^3          ▼
                         item vectors (from item tower)     features: feature store      impression log
                                  ▲                          (latest values) ◄── stream aggregates
                                  │                                                      │
 training: logs ─► point-in-time joins ─► towers + ranker ─► export ─► re-embed items ─► rebuild index

How the parts fit together

Event stream and aggregates

Every impression, play, skip, like and hide is an event. Windowed stream aggregation groups events by event-time window per key (user, item, creator) and emits a window when the watermark passes its end: "plays of item X in the last hour", "this user's skips today". Windows are slices of partial aggregates, so a sliding window costs one add and one subtract. Contract: late events are the norm (mobile clients batch their uploads), so the watermark and the allowed lateness decide how fresh the number is versus how complete. Heavy-tailed keys (every item, every search term) do not all deserve exact counters, so a Count-min sketch keeps approximate frequencies in fixed memory, with a bounded overestimate; it is used for item popularity, for "how many times has this user seen this creator", and, as below, to correct sampling bias in training.

Feature store

The Feature store gives each feature one definition that feeds two stores: a history store for training and a latest-value store for serving. Its job is point-in-time correctness: a training row for an impression at 10:03 must contain the feature values as they were at 10:03, not today's. A model trained on leaked future values scores well offline and badly online (training-serving skew). The contract with the ranker is a flat list of named features with types and a freshness class (real time, hourly, daily).

Candidate generation

Two-tower retrieval encodes users (history, context) and items (id, content, creator) into one vector space with two separate networks. Item vectors are computed offline, so serving costs one user-tower forward pass plus a nearest-neighbour search. Both towers lean on Embeddings, learned tables from ids to dense vectors. A user tower that reads a long sequence of recent items usually does so with a Transformer block over the history.

The towers are trained with Contrastive learning: the pair (user, item they engaged with) should score higher than that user against every other item, with an InfoNCE loss and a temperature. Doing the softmax over the full corpus is too expensive, so Negative sampling contrasts each positive with a few sampled negatives, and the cheapest sampler is the other items in the same batch. That sampler is biased in a power-law corpus (popular items appear as negatives far more often than rare ones), and the YouTube retrieval paper corrects it by estimating item frequency from the stream (Yi et al., RecSys 2019); a count-min sketch is the textbook alternative for that estimate. Embedding-based retrieval has the same shape outside video: Facebook's search paper describes serving it inside an inverted-index based system (Huang et al. 2020).

Matrix factorization for collaborative filtering is the older alternative: learn a vector per user and per item from the observed cells, and predict by dot product. It is simple, fast to train, and still a good baseline source, but it has no vector for an item that nobody has touched yet, and a content tower does. Real systems run several sources side by side (two-tower, follow graph, item-to-item, popular in region, fresh uploads) and merge the lists by id; X reports that about half of home-timeline posts come from its in-network source (source: twitter/the-algorithm README).

Vector index

Item vectors are searched by approximate nearest neighbour. HNSW is a layered proximity graph searched greedily and is the default in-memory index: high recall at low latency, but the graph and the full vectors must be in RAM. IVF-PQ clusters vectors into lists and stores each as a few bytes of product-quantisation codes, scanning only the nearest lists; it needs far less memory per vector at some recall cost. The choice is a memory-versus-recall trade-off at corpus size; Faiss implements both families. Top-k selection (a bounded heap) keeps the best candidates across lists and across shards.

Contract: the index and the user tower are a matched pair. Vectors from model version N are only comparable with queries from version N. A model release therefore re-embeds the corpus and builds a new index before traffic moves, and the serving layer must never mix versions.

Ranker

The ranker sees a few hundred to a thousand candidates (order of) and can afford many more features per candidate than the retrieval stage: user history aggregates, item stats, creator stats, user-item crosses, context. Click and conversion prediction here is a sparse-feature problem: feature-interaction-models (no page yet) holds the model families that learn feature crosses (DCN V2 is one: it learns bounded-degree feature interactions, Wang et al. 2020; Wide & Deep joins a memorising wide part with a generalising deep part, Cheng et al. 2016). Click-probability prediction (CTR models) and Deep ranking models for recommendation are the pages for the individual models. Id features are huge, so they pass through Feature hashing (the hashing trick) into embedding tables, which are the memory problem of this part: DLRM splits it by using model parallelism for the embedding tables and data parallelism for the dense layers (Naumov et al. 2019), and embedding-table-sharding (no page yet) is the technique. ByteDance's Monolith replaces hashing with a collisionless embedding table, with expirable embeddings and frequency filtering to bound memory, so two ids never share a vector (Liu et al. 2022).

One ranker usually predicts several outcomes (click, watch time, like, share, hide), because optimising a single click rate rewards clickbait; YouTube's multitask ranking paper treats competing objectives and implicit selection bias in user feedback as the core difficulty (Zhao et al. 2019). The heads are then combined by a weighted sum into a value. That sum multiplies probabilities, so each must be calibrated: Calibration fits a map such as isotonic or Platt scaling on held-out data so that a predicted 0.1 means about 10 % of such items get the outcome. Otherwise changing one head's training data silently re-weights the whole objective.

Re-ranker and exploration

Sorting by value gives a page of near-duplicates: ten videos from one creator, ten angles on one topic. maximal-marginal-relevance (no page yet) picks items greedily by relevance minus similarity to what has been picked. Business rules sit here too (age limits, blocked creators, per-creator caps).

Exploration is the other job. A new item has no history, so the ranker underrates it and it never collects any. Multi-armed bandits allocate a share of slots to uncertain items (UCB or Thompson sampling) so their payoff becomes known and so the data the next model trains on is not only what the current model already liked. The alternative, a fixed exploration rate, pays the same cost forever. Contract: every exploration slot logs its propensity, so training can correct for it.

Evaluation

Offline, Evaluation metrics measure retrieval (recall at k of held-out engagements), ranking (NDCG, AUC, log loss) and calibration (ECE), with paired confidence intervals so a 0.2 % gain is not noise. Offline numbers only qualify a change; they do not prove it, because the data was produced by the previous model. A/B testing (online controlled experiments) does: randomly split users into arms by a salted hash, log exposure, compare long-term metrics with a confidence interval, and guard with sample-ratio, guardrail and peeking checks. The alternative for allocating traffic while learning is the bandit; for product decisions that need a clean effect size, an A/B test is the tool.

Data and control flow

One feed request:

1. The client sends user id and context; the feed service loads the user's latest features from the feature store.
2. The user tower computes the user vector (or reads a cached one).
3. Each candidate source returns up to a few thousand ids (order of): ANN search, follow graph, popular, fresh. Merge by id and drop what was already seen.
4. The ranker scores the survivors; features are fetched in one batched lookup, because feature fetch, not the neural net, is often the largest part of ranking latency.
5. Calibrated outputs are combined into a value; hard filters run (safety, blocked, age).
6. The re-ranker applies diversity and exploration slots; the page is returned and every shown item is logged with its position, scores and propensity.

An interaction becoming a feature:

1. A play event reaches the log; the stream job slots it into the user's and item's windows.
2. Aggregates are written to the latest-value store and appended to the history store.
3. The next request, seconds later, sees the changed counts; the next training run sees the same values as of the time.

A model refresh (daily to weekly, order of):

1. Join impressions with outcomes and point-in-time features; train towers and ranker.
2. Export; re-embed the whole corpus with the new item tower; build the new ANN index.
3. Calibrate the ranker on held-out data, shadow-score live traffic, then ramp through an A/B test.
4. Swap tower, index and ranker together.

Scaling

In the order they bite:

1. Per-request compute. The funnel is the lever: each stage scores fewer items with a more expensive model. Moving the cut earlier saves the most, and costs recall.
2. Embedding-table memory. Id features for users and items outgrow one device. Levers: hashing, collisionless tables with expiry and frequency filters, sharding by row or table across devices (TorchRec's purpose, "large embedding tables sharded across many GPUs"), and on the serving side quantised or cached rows.
3. Index memory and build time. HNSW holds everything in RAM. Levers: IVF-PQ compression, sharding the corpus and merging per-shard top-k, filtering to recent or eligible items before indexing.
4. Feature fetch latency. Levers: batch lookups, co-locate features by user, cache hot users, precompute the expensive ones.
5. Freshness. Daily models miss what became popular this hour. Levers: stream aggregates as real-time features, online training of the sparse parameters (Monolith's design aim), a fresh-items source.
6. Measurement. A/B tests need traffic and time. Levers: variance reduction, many layered experiments, offline replay for screening.

Failure modes

Real implementations

Sources

• P. Covington, J. Adams, E. Sargin, Deep Neural Networks for YouTube Recommendations, RecSys 2016. Google Research
• X. Yi et al., Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations, RecSys 2019. Google Research
• Z. Zhao et al., Recommending What Video to Watch Next: A Multitask Ranking System, RecSys 2019. Google Research
• M. Naumov et al., Deep Learning Recommendation Model for Personalization and Recommendation Systems, 2019. arXiv
• Z. Liu et al., Monolith: Real Time Recommendation System With Collisionless Embedding Table, 2022. arXiv
• J.-T. Huang et al., Embedding-based Retrieval in Facebook Search, 2020. arXiv
• R. Wang et al., DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems, 2020. arXiv
• H.-T. Cheng et al., Wide & Deep Learning for Recommender Systems, 2016. arXiv
• J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with GPUs, 2017 (Faiss). arXiv
• Y. Malkov, D. Yashunin, Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs, 2016. arXiv
• X recommendation algorithm README (pipeline components, candidate sources). repo
• TorchRec README (sharded embedding tables). repo