synthetic-monitoring

How it is built

Architecture overview

Synthetic monitoring answers one question from the outside: does the service work right now, as a user would experience it? A fleet of probers in several locations runs scripted checks against a target on a fixed schedule: resolve a name, open a TLS connection, fetch a URL, or drive a real browser through a login. Every run produces a result (success flag, timings, certificate facts). A central side turns the stream of results into availability, latency percentiles and alerts. In the vocabulary of the Google SRE book this is black-box monitoring: symptom-oriented, it reports problems that are active now, not ones predicted from internal state.

The idea that makes it work is that a single failed probe proves little. A check is declared down only after retries and agreement between locations, and the checks themselves are spread over locations by a deterministic hash so that losing a prober moves only that prober's checks.

Parts:

• Check scheduler — turns check definitions (target, interval, timeout) into due probe runs, with jitter — cron-scheduling (no page yet), built on Hierarchical timing wheel
• Location assignment — decides which prober locations run which check and takes over when one disappears — Rendezvous hashing, Leases
• Protocol probes — the network-level checks: name resolution, TLS handshake, certificate validity — DNS protocol, TLS 1.3, x509-certificates (no page yet)
• Browser probes — scripted journeys in a real browser, with screenshots on failure — browser-automation-protocol (no page yet)
• Result aggregation — merges timings from all locations into latency distributions per window — quantile-sketches (no page yet), hdr-histogram (no page yet)
• Health evaluator — turns raw results into an up/down state with retries and flap control — Health checking
• Alerting — groups, deduplicates and routes notifications — alert-grouping (no page yet), built on Hierarchical timing wheel and Swiss table
• Operations and assurance — keeps probers and the control plane running and replaceable, protects check state, secures the prober link, checks the parsers — Process supervision, Graceful draining, Mutual TLS, Primary-backup replication, Point-in-time recovery, Coverage-guided fuzzing, Deterministic simulation testing

 check definitions ──► scheduler (timing wheel, jitter)
                           │ due (check, run-id)
                           ▼
              assignment: top-k locations by rendezvous hash
                           │
        ┌──────────────────┼──────────────────┐
        ▼                  ▼                  ▼
   prober eu-1        prober us-1        prober ap-1      (each holds a lease on its work)
   DNS/TCP/TLS/HTTP   same                same
   browser pool       same                same
        │ result: ok, phase timings, cert expiry, error class
        └──────────────────┬──────────────────┘
                           ▼
        result ingest ──► sketches per (check, location, window) ──► dashboards, SLO views
                           │
                           ▼
              health evaluator (retries, quorum) ──► alert grouper ──► pager / chat / webhook

There are two processes with different needs. The probers are many, small, and placed where users are; they must be cheap to run and must not need to be trusted with the whole fleet's data. The control side is few, holds the check definitions and the long-term results, and must be more reliable than anything it watches.

How the parts fit together

Check scheduler

A check is a recurring job, so its schedule is a cron-scheduling (no page yet) style or fixed-interval rule. With many thousands of checks, a naive loop that scans all of them every second is wasteful; a Hierarchical timing wheel buckets deadlines into slots so that inserting, cancelling and firing a timer are constant-time operations, and the scheduler only touches checks that are due. Two details matter for correctness more than for speed:

• Jitter. Checks that share an interval must not all fire on the same second, or the monitor itself becomes a load spike on the target (and on its own result path). The first fire is offset by a stable per-check value, for example derived from a hash of the check id, so a restart of the scheduler does not re-align everything.
• Missed runs are skipped, not replayed. After an outage of the scheduler, running every missed probe at once would measure the recovery storm. The scheduler computes the next due time from now.

Location assignment

Each check names how many locations should run it. The assignment function is Rendezvous hashing: for a check, score every live location by hashing the pair (check id, location id) and take the top k. It needs no table and no coordination, every control-plane node computes the same answer, and when a location leaves only the checks that had it in their top k move, to their next best location. A ring with virtual nodes would give a similar property; rendezvous hashing is simpler here because the location set is tiny (tens, not thousands) so scoring all of them per check is cheap.

Moving checks quickly after a failure needs an agreed view of who is alive. A prober holds a Lease on its assignment, renewed with each heartbeat; when the lease expires the control side stops counting its results and the hash simply recomputes without it. The same lease pattern protects the control side: one evaluator is leader for a check shard at a time. A duplicate probe run during a hand-over is harmless (probes are side-effect-free by design) so the system accepts at-least-once execution instead of paying for exactly-once. Browser journeys that write data, such as placing a test order, are the exception: they need their own idempotency and cleanup.

Protocol probes

These are the cheap, frequent checks, and each one splits a request into phases so a failure is attributed to a layer, not just "down".

• Name resolution (DNS protocol, RFC 1035): query a record type against a chosen resolver or directly against the authoritative servers, compare the answer with the expected value, and record query time. Probing the authoritative servers separately from a recursive resolver separates "our zone is broken" from "a resolver cache is stale".
• TLS (TLS 1.3, RFC 8446): connect, complete the handshake, and record handshake time, negotiated version and cipher suite. Failing to complete the handshake is a different alert from a slow one.
• Certificates (x509-certificates, no page yet; RFC 5280): validate the chain against a trust store, check the hostname against the subject alternative names, and export days-to-expiry as a metric. The most useful alert in this whole system is often "certificate expires in N days", because it is a failure that is certain, dated and completely avoidable.
• HTTP and TCP/ICMP: status code, redirect chain, body assertions, time to first byte. The Prometheus blackbox exporter exposes HTTP, HTTPS, DNS, TCP, ICMP and gRPC probes; its ICMP prober needs elevated privileges on the host, which is a deployment constraint for prober images.

Each probe has a hard timeout shorter than the check interval, and its result carries an error class (DNS failure, connect timeout, TLS failure, bad status, assertion failed) as a bounded label, never the raw error string, so the number of distinct series stays fixed.

Browser probes

Some failures only appear in a rendered page: a script that throws, a login form that no longer submits. A browser probe runs a script against a real browser engine through a remote-control protocol such as the Chrome DevTools Protocol (browser-automation-protocol, no page yet), which is JSON messages over a WebSocket. Playwright wraps this control for Chromium, Firefox and WebKit behind one API. Contracts to get right:

• Isolation. A fresh browser context per run, so cookies and storage never leak between runs.
• Budget. A journey has a wall-clock limit and a step limit; a hung page must not pin a prober.
• Evidence. On failure, keep a screenshot, the failing step and the network log, because an on-call engineer will otherwise have only "step 4 timed out".
• Cost. A browser run is orders of magnitude heavier than a DNS query, so browser checks run at lower frequency from fewer locations, in their own worker pool.

Result aggregation

Availability is a ratio of counts, which merges by addition. Latency is not: you cannot average the p95 of three locations to get the p95 of the union. The probers therefore send timings into mergeable summaries, quantile-sketches (no page yet), such as DDSketch, which has relative-error guarantees and merges exactly, or the t-digest. A sketch per (check, location, window) is small and fixed in size, so a one-minute window from every location can be combined into an hourly or daily one without keeping raw samples. Reporting by location also reveals regional problems that a global number hides.

Inside a prober, hdr-histogram (no page yet) is the fixed-size bucketed form for the timings it measures itself: log-linear buckets with bounded relative error give DNS, connect, TLS and first-byte percentiles for a window from a few kilobytes, and two histograms with the same layout merge by adding bucket counts. That is what is sent to the aggregator, which merges across locations and windows as above.

Health evaluator

The evaluator applies Health checking semantics to the result stream: a check flips to down after N consecutive failures (or when a quorum of its locations fails within a window) and back to up after M successes. Asymmetric thresholds avoid flapping. Two rules keep the signal honest: a failure seen from only one location, with the other locations healthy, is classified as a location problem or a regional path problem, not a service outage; and a prober that stops reporting is a no data state, never up.

Alerting

State changes feed an alert grouper (alert-grouping, no page yet). Alerts carry label sets (check, location, error class, team); a map keyed by the label set (Swiss table) deduplicates repeats, and timers on the timing wheel implement "wait a few seconds to collect related alerts, then send one notification, and remind every few hours". Prometheus Alertmanager is the reference for this: it deduplicates, groups, silences and inhibits alerts and routes them to receivers such as e-mail, PagerDuty or a webhook. Inhibition matters here: a DNS failure should mute the HTTP and TLS alerts for the same target, since they are consequences.

Operations and assurance

A check that stops running looks like a healthy target, so the monitoring system must watch itself first. Each prober runs under Process supervision, which restarts a crashed process with a delay and a burst limit; a prober that cannot stay up stops renewing its leases, and the location assignment moves its checks elsewhere. When a location is retired or updated, Graceful draining is the clean exit: it stops accepting new runs, lets running probes and browser journeys finish by a deadline, sends the last results, and releases its leases so the checks move at once, without a gap in the series.

Probers sit in many networks, some inside customers' private networks, and report to a central service. Mutual TLS gives each prober a verified, rotatable identity on that link: the control plane hands check definitions (which may hold credentials for login journeys) only to probers it enrolled, and accepts results only from them, so a rogue host cannot feed false "all green" data. The control plane's state (check definitions, current up/down state, flap counters) is protected by Primary-backup replication: one node orders the writes and ships them to a standby, the acknowledgement rule decides whether a state change that was alerted on can be lost, and an external arbiter chooses the primary so two schedulers never run the same checks as owners. Point-in-time recovery rebuilds check definitions and result history to the instant before a bad bulk edit (an interval changed to a day, checks deleted) from a base backup plus the archived change log.

Probers parse answers from servers they do not control: DNS messages, TLS certificates, HTTP responses and the pages loaded by the browser. These parsers are the inputs for Coverage-guided fuzzing, where a mutated response that reaches a new edge is kept; a crash in a prober on one hostile response would blind that location. Deterministic simulation testing runs scheduler, location assignment, probers and a simulated network and clock in one thread under a seeded scheduler, with locations that disappear, leases that expire and clocks that skew, and checks that every check runs at its interval from some location and that flap control does not alert on a single lost probe.

Data and control flow

One HTTP check, end to end

1. The timing wheel fires check C at its jittered due time; the scheduler writes a run record (check id, run id, due time).
2. The assignment function scores the live locations for C and picks the top k; each chosen prober receives the run (push) or picks it up on its next poll (pull).
3. The prober resolves the name, connects, performs the TLS handshake, sends the request and applies the assertions, recording each phase's duration; a global timeout cuts the run off.
4. The prober sends one result message: success flag, error class, phase timings, certificate not-after time.
5. Ingest adds the timings to the sketch for (C, location, current window) and the flag to a counter; the evaluator updates C's state.
6. If the state changes, an alert event with C's labels goes to the grouper. Time is dominated by the target and the network (tens to hundreds of milliseconds for a healthy HTTPS fetch, order of); the monitor's own overhead should be negligible against it.

A prober fails

1. Its lease stops being renewed; the lease expires after its term.
2. The control side drops that location from the live set; rendezvous hashing recomputes top k for the affected checks only.
3. Replacement probers pick up those checks on their next cycle; results from the failed prober that arrive late are discarded by lease epoch or run id.
4. A "prober missing" alert is raised on the monitor itself, routed by a path that does not depend on the failed prober.

An alert

1. The evaluator emits down for C with labels. 2. The grouper waits for related alerts in the same group, then sends one notification. 3. A repeat timer re-notifies until the check recovers or someone silences it.

Scaling

In the order they bite:

1. Browser probes cost most. Levers: lower frequency, fewer locations, a separate pool with a cap on concurrent browsers, and recycling browser processes after a bounded number of runs.
2. Timer and scheduling load. Millions of timers are cheap on a timing wheel; if one scheduler is not enough, shard checks by hashing the check id to a scheduler shard.
3. Result volume. Checks × locations × runs per minute. Levers: pre-aggregate in the prober into one summary per window instead of one message per run, keep raw runs short-lived, store sketches long-lived.
4. Cardinality. Labels such as the raw URL or error text explode series counts; bound them to fixed classes.
5. Prober fleet. Add locations; rendezvous hashing rebalances only a share of checks proportional to the change.
6. Alert volume. Grouping and inhibition keep human notifications flat while checks grow.

Failure modes

Real implementations

The alternative: white-box monitoring, where the service exports its own metrics and a collector scrapes them, sees causes that probes cannot (queue depth, error counts by handler) but cannot see failures that sit in front of the service: DNS, certificates, routing, a CDN. The Google SRE book recommends heavy use of white-box monitoring with modest but critical use of black-box monitoring; synthetic monitoring is that second half.

Sources

• B. Beyer et al., Site Reliability Engineering, ch. 6 "Monitoring Distributed Systems" (black-box versus white-box monitoring). chapter
• RFC 8446, The Transport Layer Security (TLS) Protocol Version 1.3. rfc
• RFC 5280, Internet X.509 Public Key Infrastructure Certificate and CRL Profile. rfc
• RFC 1035, Domain names - implementation and specification. rfc
• C. Masson, J. E. Rim, H. K. Lee, DDSketch: A fast and fully-mergeable quantile sketch with relative-error guarantees, 2019. paper
• T. Dunning, O. Ertl, Computing Extremely Accurate Quantiles Using t-Digests, 2019. paper
• Chrome DevTools Protocol. docs
• Prometheus blackbox exporter README (supported probers, ICMP privileges). repo