continuous-benchmarking

How it is built

Architecture overview

Continuous benchmarking runs performance tests on every change, stores the numbers, and tells a human or a merge gate when a change made the software slower. It is continuous integration for speed: a unit test answers "does it still work", a benchmark run answers "is it still fast", and both are checked by automation before the change reaches users (Bencher's own documentation makes this analogy explicitly). The hard part is not running the benchmark. It is that a timing is a noisy random variable, so a single number from a pull request cannot be compared with a single number from main. The system therefore has to do three things: make each measurement as repeatable as possible, keep enough history to know how much a metric normally varies, and apply a statistical rule that separates a real shift from noise.

There are two checks, and they answer different questions. The pull-request check compares a candidate against its base, fast, with a pass/fail result and a report on the PR. The trend watcher looks at the long history of main after merge and finds the commit where a metric stepped to a new level, including slow drifts that no single PR crossed a threshold for. When either one fires, an attribution step names the commit (or the small set of commits) responsible.

Parts:

• Harness — the benchmark programs and their output format (name, value, unit, repeats) — benchmarking-methodology (no page yet)
• Measurement environment — the machine and the metric: dedicated hosts, or instruction counts instead of wall time — hardware-performance-counters (no page yet), CI runner fleet, Durable job queue, Leases
• Result store — every sample keyed by benchmark, branch, testbed and measure, with its commit — Time-series chunk storage, quantile-sketches (no page yet)
• Pull-request comparison — candidate versus base, one verdict per benchmark — Hypothesis testing, Robust statistics, bootstrap-confidence-intervals (no page yet)
• Trend watcher — finds steps and drifts in the history of the main branch — change-point-detection (no page yet), Time-series anomaly detection
• Attribution — turns "it got slower sometime in these 40 commits" into one commit — failure-bisection (no page yet)
• Reporting and gating — PR comment, required status check, alert to the owner — Alerting, CI server

 commit / PR ─► CI server ─► bench job queue ─► dedicated runner (pinned, quiet)
                                                    │  run N repeats, read counters
                                                    ▼
                                          raw samples + metadata
                                                    │ upload (name, branch, testbed, measure)
                                                    ▼
                                           ┌──── result store ────┐
                      candidate vs base ◄──┤  time series per key │──► change-point scan on main
                      (test + effect size) └──────────────────────┘          │
                              │                                              ▼
                              ▼                                  suspect range ─► bisection runs
                   PR status check + comment                                 │
                   (block / acknowledge)                                     ▼
                                                            alert with the responsible commit

The idea that makes it work: decide on a distribution and a rule written down in advance, never on one number. The rule can be a threshold on a z-score, a test between two sets of repeats, or a change-point score on a long history, but it always knows how noisy this benchmark is on this machine.

How the parts fit together

Harness and measure

A benchmark harness (Google Benchmark, Criterion for Rust, Go's testing package benchmarks, pytest-benchmark) runs a function many times and prints a result. Continuous benchmarking wraps the harness rather than replacing it: Bencher's bencher run takes the harness output, parses it with an adapter, and sends a report to its server, so the team keeps the harness it already has. What matters for the rest of the system is the identity of a result. Bencher's data model is a good template: a Benchmark is a named test, a Metric is one result of it, a Measure is the unit (latency, throughput), and a Report ties a set of metrics to a Branch and a Testbed. A threshold belongs to one combination of branch, testbed and measure. If a benchmark is renamed or moved, the history is cut, because the name is the key (the Bencher docs warn about exactly this). The discipline of what to run, how many repeats and how long a warm-up are covered by benchmarking-methodology (no page yet).

Measurement environment

Noise is the first limit on how small a regression you can see, so the environment comes before the statistics. Bencher's documentation lists three options for general-purpose CI machines: bare-metal runners, relative benchmarking (run base and candidate on the same machine in the same job), and a harness that counts instructions instead of wall-clock time. It claims under 2% variance for its own bare-metal runners against more than 30% between runs on shared GitHub-hosted runners; treat those as vendor figures for the vendor's setup, not a law. CodSpeed's CPU-simulation instrument takes the instruction-count idea further: it runs a benchmark once and simulates the CPU, estimating cycles from executed instructions plus L1 and last-level cache misses, so the result is repeatable across machines; its documentation recommends it as the default and tells you to switch to its wall-time instrument, which runs on bare-metal runners, when a benchmark does I/O or is too long to simulate. The Rust project's rustc-perf takes a similar practical line: its glossary gives instruction count as the example metric. Linux exposes such counters through perf_event_open (man page); hardware-performance-counters (no page yet) covers what they do and do not capture.

The contract between runner and the rest: a runner is one job at a time, nothing else on the host, and every sample carries the testbed name so that results from different hardware never get mixed in one series. Scheduling those exclusive jobs is the ordinary pattern of a Durable job queue with Leases: an agent takes a job under a lease, and if the host dies the job returns to the queue (see CI server and CI runner fleet for the general machinery).

Result store

The store holds, per key, an ordered series of samples with the commit hash and time. It is small by database standards (a few numbers per benchmark per commit) and read in two shapes: the last N points of a key for a threshold test, and the whole series for a change-point scan. A plain relational table works for most teams; at large volume a time-series layout with delta and XOR compression (Time-series chunk storage, Gorilla XOR compression) keeps it cheap. For latency benchmarks, storing one number per run throws away the tail. A mergeable summary of the whole latency distribution per run (quantile-sketches, no page yet) lets you compare p99 between runs and combine repeats.

Pull-request comparison

Two styles are in use. A threshold on history compares the new value with the recent past of the same key: Bencher's tests are percentage, z-score, Student's t-test, log-normal, interquartile range, delta-IQR and static, each with a lower and/or upper boundary, a minimum and maximum sample size and a window; an alert fires when the new metric is beyond the boundary limit. Rust's rustc-perf uses an outlier rule of the same family: the relative change of a test is significant if it exceeds the third quartile of historical relative changes by a multiple of the interquartile range (the formula in its docs uses 3), and it then combines many tests into one "improvement, regression or mixed" verdict. A two-sample test between repeats compares candidate repeats with base repeats directly: Google Benchmark's compare.py uses the Mann-Whitney U test with the null hypothesis that there is no difference in performance, and Go's benchstat computes statistical summaries and A/B comparisons of before/after benchmark files. Medians and MAD are the usual summary here because one slow repeat should not move the baseline (Robust statistics). The general reasoning about p-values, effect size and many simultaneous comparisons is in Hypothesis testing; when the distribution is awkward, a bootstrap confidence interval for the ratio of medians (bootstrap-confidence-intervals, no page yet) avoids assuming a normal shape.

The practical rule that keeps this useful: test for a minimum effect size, not only for "statistically different". With enough repeats a 0.3% shift is significant and still irrelevant.

Trend watcher

Per-PR checks miss small steps that stay below the threshold and slow drifts. The trend watcher runs after merge on the full history of main. MongoDB's team describes replacing manual graph inspection and then thresholds with change-point detection based on the E-Divisive means algorithm: it returns a list of change points from the history, a human triages the list, and they report a dramatic drop in false positives (Daly et al., see Sources). The open-source Hunter tool from DataStax describes a modified E-Divisive means that replaces the randomized permutation significance test with a Student's t-test, because the randomized version gave non-deterministic results at a reasonable number of iterations; Hunter's repository now carries the README of Apache Otava, which is listed below. change-point-detection (no page yet) is the block; Time-series anomaly detection covers the simpler single-point view.

Attribution

If a change point lands between two nightly runs, the candidate range can contain many commits. Bisection reruns the benchmark on the midpoint commit, narrows the range and finishes in about log2 of the commit count runs. git bisect (docs) provides the loop; failure-bisection (no page yet) is the block. The catch for performance, unlike a crash, is that "good" and "bad" are statistical: each probe must run enough repeats on the same testbed, or the search walks down the wrong half.

Reporting and gating

The result of a PR run becomes a comment and a status check. CodSpeed compares the branch with main against a project regression threshold and can fail the check, blocking the merge unless the regression is fixed or explicitly acknowledged. Whatever the tool, keep an "acknowledge" path with a recorded reason: some regressions are intended (a safer algorithm that is slower), and a gate without that path gets turned off. Alerts from the trend watcher go to the owning team through the normal Alerting route.

Data and control flow

One pull request:

1. The CI server receives the PR event and enqueues a benchmark job (build, then run); a free dedicated runner takes it under a lease.
2. The runner builds base and candidate, or reuses a cached base result for that commit, and runs each benchmark with a warm-up and N repeats, reading counters or timers. Time goes here: build time plus (benchmarks x repeats x run time) is the whole cost of the check.
3. Samples are uploaded with branch, testbed, measure and commit.
4. The comparison reads the last window of the key on main (or the base repeats), runs the test, applies the minimum effect size and writes verdicts.
5. The status check and PR comment are posted. A failing verdict blocks the merge until fixed or acknowledged.

Nightly trend scan:

1. A scheduled job reads the full series of every key on main.
2. The change-point routine returns candidate steps with a score; steps already reported are dropped.
3. For each new step, the attribution job bisects the commit range on the same testbed.
4. An alert names the commit, the metric, the size of the shift and a link to the graph.

Scaling

Pull the levers in this order:

1. Noise before volume. More repeats cost time linearly and shrink noise only as the square root, so first fix the environment: dedicated host, pinned frequency, an instruction-count metric where it fits. This is what lets a smaller effect be detected at all.
2. Run less per change. Run only the benchmarks whose code changed, shard the suite across runners, and run the expensive end-to-end set nightly instead of per PR. CodSpeed's docs list sharded benchmarks and partial runs as features for this reason.
3. Cheaper history. Keep raw samples for a window and roll up older data to per-commit summaries; use the compressed series layout if the store grows.
4. Fewer false alarms as metrics grow. Hundreds of benchmarks times several measures means many tests per PR, so some will trip by chance. Require the minimum effect size, require the effect on more than one repeat set, and aggregate (rustc-perf's "mixed" verdict) before blocking.
5. Bisection cost. Each probe is a full build plus a run; cache build outputs per commit and bisect only on the one metric that moved.

Failure modes

Real implementations

The alternative: do not store history; run candidate and base back to back on the same host in the same job and compare only those two (relative continuous benchmarking). It needs no result store and cancels most machine drift, but it cannot see slow drift, cannot alert after the fact, and doubles the run time of every PR. Most teams use relative runs for the PR gate and a stored history for the trend watcher.

Sources

• Bencher documentation: what is continuous benchmarking, benchmarking overview, thresholds and alerts.
• CodSpeed documentation: instruments, CPU simulation, performance checks.
• D. Daly, W. Brown, H. Ingo, J. O'Leary, D. Bradford, Change Point Detection in Software Performance Testing, 2020. arXiv:2003.00584
• M. Fleming et al., Hunter: Using Change Point Detection to Hunt for Performance Regressions, 2023. arXiv:2301.03034
• rustc-perf: comparison analysis and the repository README.
• Google Benchmark: tools and compare.py. Go: benchstat.
• Linux: perf_event_open(2). Git: git-bisect.