model-monitoring

How it is built

Architecture overview

A deployed model degrades without any code change, because the world it was trained on moves: inputs shift, the relation between inputs and outcomes changes, an upstream feed breaks. Model monitoring is the system that notices. It records what the model saw and predicted, summarises those records per time window into small statistical profiles, compares each window with a reference (the training data or a healthy past period), and joins the predictions with ground-truth labels when those eventually arrive to compute real quality. Alerts fire when a distribution, a drift score or a quality metric leaves its expected band.

The one idea that makes it work is that labels are late, the inputs are not. Quality (accuracy, error) needs the outcome, which can arrive hours, weeks or never; so the system watches things that are available immediately (input distributions, prediction distributions, confidence) as early-warning proxies, and computes the true metric whenever a label is joined in. The data volume is large, so the profiles are built from mergeable sketches of fixed size, not from stored raw rows.

Parts:

• Prediction log — one record per prediction: id, time, features, output, model version; labels joined later by id — Columnar storage
• Windowed profiler — per window, per feature summaries that can be merged across windows and shards — Windowed stream aggregation, Quantile sketches (no page yet), HyperLogLog, Count-min sketch
• Drift detector — compares a window's profile with the reference and scores the difference — Distribution drift detection (no page yet), Hypothesis testing
• Trend watcher — detects a sustained shift in a metric series, not a single noisy point — Change-point detection (no page yet)
• Embedding monitor — drift for text, image and other unstructured inputs — embedding-inference (no page yet), PCA and SVD
• Quality evaluator — metrics and calibration from joined labels, per slice — Evaluation metrics, Calibration
• Alerting and UI — thresholds, baselines, dashboards, drill-down to rows (not covered by a block)

 model service ──(id, time, features, prediction, version)──► log stream ──► columnar store (raw rows, TTL)
                                                                  │
                                         window (tumbling, e.g. 1 h) ──► profiler: sketches per feature
                                                                  │          (quantiles, distinct, top-k)
              reference profile (training data / healthy week) ──┤
                                                                  ▼
                              drift detector: per-feature score, prediction-drift, embedding-drift
                                                                  │
 labels (late, by id) ──► join ──► quality evaluator: metrics, calibration, per slice
                                                                  │
                                       metric series ──► change-point / threshold ──► alerts, dashboard

The stream path (profiles, drift) is cheap and fast and gives early warnings. The batch path (label join, quality) is slower and gives the truth. The system's value is in running both and showing where they disagree.

How the parts fit together

Prediction log

Everything starts from a faithful record. Each prediction logs a stable prediction id, a timestamp, the model version, the input features (or a reference to them), the output and its confidence. The id is the join key for labels that arrive later, so losing it makes quality monitoring impossible; logging the model version makes before/after comparisons across deployments possible. Raw rows go to Columnar storage: monitoring queries read a few columns across many rows (the distribution of one feature over a week), which is the case a column layout compresses and scans well. Rows are kept for a limited time for drill-down; the sketches below are kept long-term. The log should record inputs as the model received them, after preprocessing, as well as raw inputs where privacy rules allow, because a break in the preprocessing code is a common cause of drift.

Windowed profiler

A profile is a per-window summary of each feature. Windowed stream aggregation cuts the stream into windows by event time (when the prediction happened, not when it arrived) and handles late and out-of-order records. Within a window, each feature gets a summary chosen by its type:

• numeric features: a quantile sketch (Quantile sketches (no page yet)), which answers "what are the median and the 99th percentile" and can be turned into a histogram; t-digest (Dunning and Ertl) and DDSketch (Masson et al.) are two published designs, and both merge;
• categorical features: Count-min sketch for frequencies and heavy hitters, and HyperLogLog for the number of distinct values (a sudden jump in distinct ids is an upstream bug);
• all features: counts, null counts, min and max.

The property that matters is mergeability: sketches from two shards or two windows combine into the sketch of their union, so hourly profiles roll up into daily and weekly ones, and a map-reduce over shards gives the global view without moving raw rows. The cost is a bounded error (relative error for DDSketch, a rank error for KLL-type sketches), which is acceptable for drift scores but means a sketch is not an audit record.

Drift detector

Distribution drift detection (no page yet) is the comparison itself: two samples, a reference and a current window, and a number saying how different they are. Two families:

• Statistical tests (Hypothesis testing): two-sample Kolmogorov-Smirnov for numeric features, chi-squared for categorical ones. They return a p-value. The catch is sample size: with very large windows any tiny, harmless difference is "significant".
• Distance measures: Wasserstein distance, Jensen-Shannon divergence, KL divergence, the population stability index (PSI). They return a size of difference that you compare with a chosen threshold.

Evidently's documented defaults show the practical resolution: for reference sets up to 1000 rows it uses the KS test (numeric) and chi-squared (categorical) with a 0.05 p-value cutoff; above 1000 rows it switches to Wasserstein and Jensen-Shannon with a threshold of 0.1 (Evidently docs, see Sources). PSI is a convention from credit-risk scoring, usually read with rules of thumb of about 0.1 (small shift) and 0.25 (large shift); these are conventions, not statistical thresholds.

A second trap is many features at once. Test 200 features at the 5 percent level and about ten will fire by chance alone, so alert on the share of drifted features, correct for multiple tests, or weight by feature importance. Rabanser et al. (2018) study this setting directly, comparing two-sample tests combined with dimensionality reduction on several datasets. Beyond input drift, monitor prediction drift (the distribution of the output) and confidence drift: both are label-free, cheap, and often move first.

Trend watcher

A daily drift score is a noisy series. Change-point detection (no page yet) tells a sustained shift from a spike: methods in the CUSUM and Page-Hinkley family accumulate small deviations from the expected level and fire when the sum crosses a bound, so a slow drift triggers earlier than a fixed threshold would and a one-hour outlier does not trigger at all. Seasonality (weekday versus weekend, month-end) must be removed or compared like-with-like (this Monday against recent Mondays), otherwise the detector alarms on the calendar.

Embedding monitor

Text, images and audio have no handful of numeric features to test, so the monitor uses their embeddings from a pretrained model (embedding-inference (no page yet), run at serving time with no training): a fixed vector per input. Three methods are common. Reduce the vectors with PCA and SVD to a few components and test each component as a numeric feature. Compare a distance between the reference and current distributions (for example between centroids, or a sliced distance). Or train a domain classifier to tell reference from current: if it cannot do better than chance (AUC near 0.5), nothing detectable has shifted; if it can, its most confident examples are the drifted inputs, which makes them directly inspectable. The embedding model itself must be frozen and versioned, or its own change shows up as drift.

Quality evaluator

When labels arrive they are joined to predictions by id, and Evaluation metrics are computed per window and per slice (country, device, customer segment): an aggregate that looks stable can hide one segment collapsing, so the evaluator keeps metrics for the top slices and uses the distinct and heavy-hitter sketches to find which slices exist. Calibration is monitored separately: a classifier can keep its ranking quality (AUC) while its stated probabilities stop matching reality, which matters wherever the probability is used for a decision threshold or a price. Guo et al. (2017) showed that modern neural networks are often miscalibrated; the reliability diagram and expected calibration error (ECE) are the standard checks.

Two issues dominate. Label delay: the metric for last week is only complete next month, so dashboards must show the label coverage of each window and never compare a complete old window with an incomplete new one. Label bias: if labels exist only for cases a human reviewed, the metric describes that subset. Some tools therefore also estimate performance without labels from the model's own confidence (NannyML's documented approach).

Data and control flow

One prediction, from request to alert.

1. The service predicts and writes a log record (id, time, version, features, output) to the stream. Cost: a non-blocking append, off the request path.
2. The profiler assigns the record to its event-time window and updates the sketches of each feature (constant time and memory per record).
3. When a window closes (after an allowed lateness), its profiles are stored and merged into the hourly, daily and weekly rollups.
4. The drift detector compares each feature's profile with the reference and writes scores, plus prediction and embedding drift.
5. Scores enter a metric series; the trend watcher and thresholds decide whether to alert, with the reference and the changed features attached.

One label arriving.

1. A label with a prediction id arrives, days later.
2. The join finds the logged prediction (a keyed lookup in the store) and emits (prediction, label, slice keys).
3. The quality evaluator updates the metrics of the window the prediction belongs to, not the window of arrival, and bumps that window's label coverage.
4. Alerts on quality use only windows above a minimum label coverage.

Scaling

In the order the limits appear:

1. Log volume. Raw rows grow with traffic. Levers: sample the raw log (keep all rows for errors and rare segments), keep full detail only for a short TTL, keep only sketches long-term.
2. Per-feature cost. Hundreds of features times windows times slices. Levers: fixed-size sketches, compute profiles incrementally and only for the features that matter, aggregate at several time grains by merging.
3. Slice explosion. Metrics per slice can create millions of series. Levers: monitor only the heavy hitters, alert on the worst slices, not all.
4. False alarms. More features and slices mean more chance alerts. Levers: multiple-test correction, share of drifted features, change-point detectors, effect-size thresholds instead of p-values.
5. Reference management. A reference that is a year old flags every seasonal pattern. Lever: rolling reference plus a fixed training reference, compared with both.

Failure modes

Real implementations

Sources

• S. Rabanser, S. Günnemann, Z. Lipton, Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift, 2018. paper
• C. Guo, G. Pleiss, Y. Sun, K. Weinberger, On Calibration of Modern Neural Networks, ICML 2017. paper
• T. Dunning, O. Ertl, Computing Extremely Accurate Quantiles Using t-Digests, 2019. paper
• C. Masson, J. Rim, H. Lee, DDSketch: A Fast and Fully-Mergeable Quantile Sketch with Relative-Error Guarantees, VLDB 2019. paper
• Evidently documentation: data drift methods and defaults.
• Apache DataSketches: datasketches.apache.org. Arize docs: docs.arize.com.