realtime-personalization

How it is built

Architecture overview

Real-time personalization picks, per request, which variant of something a person sees: which artwork for a title, which row first on a home screen, which banner, which discount, which message. The request budget is tens of milliseconds, the catalogue of variants is small to medium (a handful of images per title, thousands of items per row), and the signal is sparse: most people click nothing most of the time. The system therefore splits the work in three speeds. A slow loop trains models and embeddings from logs every few hours or days. A fast loop turns the last seconds of behaviour into counters and features. A per-request path reads those features, scores a short list of variants, and sometimes picks one that is not the top score so the system keeps learning. The idea that makes it work is that every decision is logged together with the probability that the policy chose it. With those probabilities you can later estimate how a different policy would have done from old logs (off-policy-evaluation, no page yet) and you can correct the bias that selecting only the winners would otherwise bake into the training data.

Netflix has described the artwork case publicly: the title is fixed, the images are candidates, and a contextual bandit chooses among them per member (Netflix Tech Blog, "Artwork Personalization at Netflix", 2017; not re-verified here, see Sources). The same skeleton serves a news feed (Li et al. 2010 used a bandit for the Yahoo! Front Page Today Module), an email subject line or a pricing page.

Parts:

• Event stream and aggregates — impressions, plays, clicks become per-user and per-item windows and counters — Windowed stream aggregation, watermarks (no page yet), Count-min sketch, Hierarchical timing wheel, Change data capture, Write-ahead log, Consumer offsets
• Feature store — one definition of each feature for training (history) and serving (latest value) — ML feature store, point-in-time-join (no page yet), LSM-tree, Sort-merge join, Time partitioning, columnar-file-format (no page yet), Slab allocator
• Retrieval — when the variant set is large, a shortlist per user — Two-tower retrieval, Embeddings, Negative sampling, Contrastive learning, HNSW, SIMD programming, d-ary heap
• Scoring — probability of a click or play for each shortlisted variant — Linear and logistic regression, feature-interaction-models (no page yet), Feature hashing, Calibration, Loss functions, AdamW, L-BFGS, LU and Cholesky, Reverse-mode autodiff
• Policy and exploration — decide which variant to show, with a known probability — Multi-armed bandits, Bayesian rate estimation, PRNGs
• Experimentation — prove a new policy beats the old one — A/B testing, Deterministic bucketing, Experiment layering, CUPED, Hypothesis testing, Sequential probability ratio test, sequential-hypothesis-testing (no page yet)
• Evaluation from logs — estimate a candidate policy before exposing anyone — off-policy-evaluation (no page yet), importance-sampling (no page yet), bootstrap-confidence-intervals (no page yet)
• Serving caches and logs — keep decisions fast and the exposure log durable — W-TinyLFU, Fast hash functions, Segmented log, CRC32C checksums, fsync, O_DSYNC and O_DIRECT, Fencing tokens

 client ─► edge/API ─► personalization service ───────────────────────────► response
 (user id,  (context)  │ 1 read features (latest)   ┌ feature store (online: latest values)
  context)             │ 2 retrieve shortlist ◄─────┤ ANN index of item vectors
                       │ 3 score each variant       └ model weights (bandit / CTR model)
                       │ 4 policy picks, p = prob.
                       ▼
                 exposure log: user, context, shown, p, time ──► stream ──► windowed aggregates ─┐
                                                                    │                            ▼
 client events (click, play, dwell) ────────────────────────────────┴──► feature store (history) ┘
                                                                         │ point-in-time joins
 slow loop: history ─► train towers + scorer ─► offline replay (OPE) ─► A/B test ─► roll out

How the parts fit together

Event stream and aggregates

The behaviour a model should react to within seconds ("watched three comedies in the last hour", "ignored this banner twice") comes from Windowed stream aggregation: events grouped by key and event-time window, one partial aggregate per slice so a sliding hour is not recomputed from scratch. Late events are handled by watermarks (no page yet), which decide when a window is final. For huge key spaces (every item, every query) a Count-min sketch keeps approximate popularity in fixed memory; a Hierarchical timing wheel expires per-key state cheaply. Where the source of truth is a transactional database (account tier, purchases), Change data capture reads its Write-ahead log and replays row changes into the same stream, and Consumer offsets make every consumer restartable without loss or double counting. The contract to the next part: events carry a user id, an item id, an event time, and the exposure id of the decision that caused them.

Feature store

The most common failure of a personalization system is silent: training saw one definition of a feature and serving computes another. The ML feature store fixes it with one definition feeding two stores, a history store for training and a latest-value store for serving. Training examples are built with a point-in-time-join (no page yet): for each logged decision, take the feature values that were known at that moment, not today's. Without that, the model learns from the future and looks great offline and mediocre online. The history store is a lake of time-partitioned files (columnar-file-format, no page yet) joined by Sort-merge join; the latest-value store is a key-value store, usually an LSM-tree, or a fixed-slot in-memory table (Slab allocator) when reads must stay under a millisecond. The serving contract: a read by user id returns a bounded feature vector in a few milliseconds, and a missing user returns defaults, never an error.

Retrieval

When the variant set is a few images per title, retrieval is unnecessary: score all of them. When it is a catalogue (a row of recommended titles, a product grid), Two-tower retrieval cuts it to a shortlist: one encoder for the user and context, one for the item, both ending in Embeddings in one space, trained with Contrastive learning against sampled negatives. Item vectors are computed offline and indexed with HNSW, which finds the nearest few hundred in a fraction of a millisecond per query; distance computations run on SIMD lanes, and the best k are kept with a d-ary heap. The alternative is Matrix factorization for the shortlist, cheaper and weaker on cold items; the full retrieve-rank-rerank funnel is the recommender system page, which this page deliberately does not repeat. The contract: the shortlist is a few hundred ids with scores, and the index is rebuilt whenever the item tower changes, never mixed across versions.

Scoring

For each shortlisted variant the scorer returns the probability of the outcome that matters. The baseline is logistic regression on hashed features (Feature hashing), fitted with L-BFGS offline or online SGD; it is fast, transparent and hard to beat on small data. When data is rich, feature-interaction-models (no page yet; the family of DCN and DeepFM) learn feature crosses, trained with AdamW using Reverse-mode autodiff under a log loss. The scorer's output must be a real probability, because the policy and the business rules compare and combine it with other numbers, so a Calibration step (isotonic or Platt, checked by expected calibration error) sits on top. The contract: scores are calibrated probabilities per (user, context, variant), versioned together with the feature definitions they were trained on.

Policy and exploration

This is the heart of the page. A pure "show the highest score" policy never learns about variants it currently rates low, and the data it generates is biased toward its own choices. A Multi-armed bandit keeps exploring in proportion to its uncertainty. For a handful of arms with few features, Thompson sampling over a Beta posterior per arm (Bayesian rate estimation) is two counters per arm and one random draw (PRNGs). For per-user decisions, LinUCB (Li et al. 2010) keeps a small linear model per arm and adds an uncertainty bonus; its matrix inverse is maintained with rank-1 updates or a Cholesky factor. The policy returns the chosen variant and the probability with which it was chosen, which is written to the exposure log. That logged propensity is the contract with everything downstream. The trade-off against fixed A/B arms is made explicit in the alternatives list: a bandit wastes less traffic on losers but makes clean measurement harder, so most teams use bandits to pick within a variant and A/B tests to judge the whole system.

Experimentation

Every policy or model change ships behind an A/B test. Assignment is by Deterministic bucketing: hash the user id with a per-experiment salt (Fast hash functions), so the same person always sees the same arm with no state to store. Experiment layering lets independent tests overlap without sharing users in a way that confounds them. CUPED uses pre-experiment behaviour to shrink variance and shorten tests, which matters because personalization effects on a metric are usually small, and Hypothesis testing with a sequential design (sequential-hypothesis-testing, no page yet) lets you stop early without cheating on the error rate. The contract: exposure is logged at the moment the user could have seen the variant, not when they clicked.

Evaluation from logs

Before a new policy touches users, estimate it from the exposure log. off-policy-evaluation (no page yet) uses the logged propensities: inverse-propensity weighting reweights each logged outcome by (new policy probability / logging probability) (importance-sampling, no page yet), and the doubly robust estimator of Dudik et al. (2011) adds a reward model to cut variance. The estimates are noisy, so intervals come from bootstrap-confidence-intervals (no page yet). Offline evaluation is a filter that discards obviously bad policies cheaply; the A/B test remains the arbiter.

Serving caches and the exposure log

Two small mechanisms keep the path fast and the data safe. Decisions and features for hot users and items are cached with W-TinyLFU, whose Count-min admission filter keeps one-off keys from evicting the hot set (alternatives: LRU and Clock, S3-FIFO). The exposure log, which is the training set of the next model, is written as a Segmented log with CRC32C on every record and an explicit fsync policy: losing a few seconds on a crash is fine, silent corruption is not. When more than one process could write the model version or the experiment config, Fencing tokens stop a stale writer from overwriting a newer one.

Data and control flow

One request (budget: tens of ms, order of):

1. The client sends user id and context (device, time, surface).
2. The service reads the latest features for the user from the online feature store and, if the set is large, queries the ANN index with the user vector for a shortlist. Time goes to the two network hops and the index query.
3. The scorer computes a calibrated probability per variant; the policy turns scores and uncertainty into a choice and a propensity (for Thompson sampling, the probability that arm is the best draw, estimated or computed).
4. The response goes out; the exposure record (user, context features or a pointer to them, shown variant, propensity, model version, experiment arm) is appended to the log.
5. Later events (click, play) arrive on the same stream, keyed by exposure id.

One learning cycle (minutes to days):

1. The stream job updates windows and counters; the online feature store receives new latest values.
2. The batch job joins exposures to outcomes with point-in-time features and writes the training set.
3. The trainer fits scorer and towers; the item tower output is re-embedded and a new index is built.
4. Off-policy evaluation ranks candidate policies on the logged data.
5. The best candidate runs as an A/B arm; if it wins on the primary metric without hurting guardrails, it becomes the new default, and the bandit's priors are carried over or reset deliberately.

Scaling

In the order they bite:

1. Per-request latency: dominated by feature reads and any network round trip. Levers: co-locate the feature store, batch reads, cache hot users (W-TinyLFU), shrink the model, precompute per-user scores for small variant sets.
2. Sparse signal: clicks are rare, so learning is slow. Levers: share statistical strength with embeddings and hashed features, use priors from similar users or titles, aggregate outcomes over time windows, CUPED to cut measurement variance.
3. Index and embedding refresh: new items need vectors before they can be shown. Levers: content-based cold-start vectors, an exploration slot reserved for new items, incremental index insertion.
4. Stream and log volume: one exposure per item per request can be larger than the clicks by orders of magnitude. Levers: log only the shown slate plus propensities, sample non-clicked exposures with known weights, partition by time and user.
5. Experiment capacity: traffic is the scarce resource. Levers: layering, variance reduction, sequential stopping, off-policy screening so only promising candidates take traffic.

Failure modes

Real implementations

Netflix, Yahoo! and other operators run internal systems of this shape; their code is not public, only the descriptions in papers and blog posts.

Sources

• L. Li, W. Chu, J. Langford, R. E. Schapire, A Contextual-Bandit Approach to Personalized News Article Recommendation, WWW 2010 (LinUCB). paper
• A. Agarwal et al., Making Contextual Decisions with Low Technical Debt, 2016: the Decision Service, which logs exploration data and propensities and closes the loop. paper
• M. Dudik, J. Langford, L. Li, Doubly Robust Policy Evaluation and Learning, ICML 2011. paper
• Y. Saito et al., Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation, 2020. paper
• Netflix Technology Blog, Artwork Personalization at Netflix (2017) and Selecting the best artwork for videos through A/B testing (2016).
• Repos: Vowpal Wabbit, Feast, Open Bandit Pipeline.