browser-test-grid

How it is built

Architecture overview

A browser test grid lets hundreds of end-to-end tests drive real browsers at the same time, on machines the test author never sees. A test client (Playwright, Selenium, anything that speaks a browser-automation protocol) asks the grid for a session; the grid finds a free slot on one of its worker machines, starts or reuses a browser there, and then becomes a transparent pipe between the client and that browser until the test ends. The grid has three jobs: admit (queue session requests and match them to capacity), host (run each browser in a box with bounded CPU and memory), and clean up (take the session back when the client dies, and hand back video, screenshots and logs). The one idea that makes it work is that a session is a lease: the browser is owned by one client for a bounded time, renewed by activity, and everything about the browser (profile, cookies, process) is destroyed or reset at the end. Open examples are Selenium Grid and Playwright's remote server mode.

Parts:

• Front door — terminates client connections and routes each request to the right place — Reverse proxy, WebSockets, Consistent hashing, Mutual TLS
• Session protocol — what the client says to the browser — browser-automation-protocol (no page yet), WebSockets
• Admission queue — holds session requests until a slot exists — Durable job queue, Exponential backoff with jitter
• Scheduler — matches a request to a node and slot — Bin packing, Leases
• Capacity — the pool of browsers ready before anyone asks — warm-pool-provisioning (no page yet)
• Browser host — one isolated sandbox per session on a worker node — Cgroups and namespaces
• Artifacts — video, screenshots and visual comparison — screen-capture (no page yet), perceptual-image-diff (no page yet)
• State and operations — where the session map lives and how nodes come and go — Primary-backup replication, Point-in-time recovery, Process supervision, Graceful draining, hdr-histogram (no page yet)
• Self-test — the scheduler and proxy are tested without browsers — Deterministic simulation testing, Coverage-guided fuzzing

 test runner (shard 1/4 …)            ┌───────────────────── grid ─────────────────────────────┐
  ├ worker ─ new session ──mTLS──────►│ front door ──► admission queue ──► scheduler           │
  ├ worker ─ new session              │     │   (FIFO, timeout)        (slot match, bin pack)  │
  └ worker ─ …                        │     │                               │ lease            │
        ▲                             │     │ session map: id → node        ▼                  │
        │  commands / events          │     ▼                       ┌── node 1 ─────────────┐  │
        │  (JSON over HTTP or         │ WebSocket pipe ◄───────────►│ slot: sandbox+browser │  │
        │   WebSocket)                │ (opaque bytes)              │ slot: sandbox+browser │  │
        └─────────────────────────────┤                             │ warm slot (idle)      │  │
                                      │ artifacts: video, logs ◄────┴───────────────────────┘  │
                                      └──────────────────────────────────────────────────────────┘

How the parts fit together

Session protocol

The client speaks one of three protocols, and the grid is built around their shared shape: a session is created by an HTTP request, then commands flow one way and events flow back. The Chrome DevTools Protocol is "a JSON-based protocol that connects debug and automation targets with trusted debug and automation clients", divided into domains such as DOM and Network with commands and generated events (source: Chrome DevTools Protocol viewer). browser-automation-protocol (no page yet) is the block that covers CDP and WebDriver BiDi, a W3C Editor's Draft that defines a bidirectional protocol for remote control of browsers, next to classic WebDriver's request-response HTTP (source: W3C WebDriver BiDi draft, W3C WebDriver Level 2). Playwright has its own protocol between its client and a server it launches: browserType.connect attaches to a browser started with launchServer over a WebSocket endpoint, and the major and minor version of client and server must match (source: Playwright docs, BrowserType).

Two consequences shape the grid. First, a trust boundary: the CDP documentation says clients are assumed to be trusted because the protocol exposes high-privilege capabilities, so a raw browser WebSocket must never face the internet; the grid authenticates the caller before it hands over a pipe. Second, a version contract: the grid must either run the browser versions the client expects or reject early; Playwright's docs say a mismatched version in the server image leaves the client unable to locate browser executables (source: Playwright docs, Docker).

Front door

The front door is a Reverse proxy with one unusual duty. For ordinary requests it forwards, retries safely and applies timeouts; for a session it must also carry a long-lived WebSocket to exactly one upstream, relaying frames without interpreting them. Keeping the pipe opaque means the grid supports every protocol version the browser supports and never has to parse CDP. In Selenium Grid's Router, a new-session request goes to the queue; a request for an existing session asks the Session Map for the node and is forwarded directly (source: Selenium docs, Grid components).

The Session Map is the state that makes routing possible. With several front-door replicas, either keep that map in a shared store, as Selenium does, or avoid it by Consistent hashing of the session id to the replica that owns the session, so that a membership change moves only about 1/n of the sessions. The shared map is simpler; hashing removes a dependency from the hot path. Client-to-grid and grid-to-node links use Mutual TLS: a node that joins must prove a certificate, and a client without one cannot reach the sandbox it would control.

Admission queue

When all slots are busy, new session requests wait. Selenium holds them in a FIFO queue with a timeout and a retry interval: by default a request waiting longer than 300 seconds is rejected, the timeouts are checked every 10 seconds, and a retry interval (15, documented in milliseconds) applies when slots are busy (source: Selenium docs, CLI options). That is an in-memory queue, which is right when a lost request is only a failed test that its runner retries. Use a Durable job queue when the unit of work is a whole test job (a shard of a suite) that must survive a controller restart: workers lease a job for a visibility timeout, acknowledge it, and the job reappears if the worker dies. The job queue feeds runners; the session queue feeds the grid.

Clients that are rejected should come back with Exponential backoff with jitter, not in a synchronised wave. The alternative, rejecting at once when no capacity exists, is useful only when nothing can scale up on demand (Selenium has a --reject-unsupported-caps option for the case where the grid cannot start new nodes for an unsupported capability, source: Selenium docs).

Scheduler

The scheduler decides which node and slot gets the session. The first filter is capability matching: browser name, version, platform. The second is placement. This is Bin packing: sessions are items, node slots are bins, and with identical sessions the interesting question is spread versus pack. Selenium offers a GreedySlotSelector as an alternative to its default; it "aims to maximize node utilization by concentrating sessions on one node before using another", which leaves whole nodes idle and therefore removable when scaling down (source: docker-selenium README). Pack when nodes can be scaled in; spread when one noisy session must not affect many neighbours.

The scheduler hands out a Lease: the client owns the slot as long as it keeps the session active, and the node kills a session with no activity for the session timeout (300 seconds by default in Selenium; source: Selenium docs). The lease makes crashes safe: a runner that dies mid-test cannot hold a browser forever. It also needs fencing: after expiry the old session id must stop working before the slot is reused, or a zombie client could drive another test's browser. A new context or container per session, with an unguessable id, gives that for free.

Capacity

Starting a browser takes time, and in a container grid it takes more: Selenium waits up to 55 seconds by default for a browser server to start in a container, 120 in a Kubernetes pod (source: Selenium docs, CLI options). warm-pool-provisioning (no page yet) hides that cost by keeping pre-initialised browsers ready, sized from the arrival pattern of session requests. Two warm-pool granularities exist, with different isolation:

• Warm browser, fresh context. One browser per slot is kept; each test gets a new context, which Playwright calls "fast and cheap to create" and isolated even within one browser (source: Playwright docs, Isolation). A crashed browser takes its slot's other work with it.
• Warm container, fresh session. A sandbox is created per session and destroyed afterwards, as Selenium's --drain-after-session-count option supports for Kubernetes (a node shuts down after a set number of sessions). Slow to create unless pre-warmed, but nothing leaks between sessions.

Browser host

On each worker, a browser needs a box: its own filesystem view, process table, network, and a budget. This is Cgroups and namespaces in the form of a container: namespaces hide other sessions' processes and files, cgroups cap memory and CPU so one runaway page cannot starve its neighbours. Selenium's default node creates one slot per available CPU for Chromium-based browsers and Firefox (source: Selenium docs), and its max-sessions option defaults to the number of processors, with a warning that overriding it can hurt stability because "the host could run out of resources". Playwright's Docker guide recommends --ipc=host because without it "Chromium can run out of memory and crash", --init to avoid zombie processes under PID 1, and a seccomp profile with a non-root user when running untrusted sites (source: Playwright docs, Docker). Your own tests can accept the root-user default; arbitrary URLs cannot.

Artifacts

screen-capture (no page yet) records the virtual display (damage regions, then encoding); Selenium's Docker images ship a separate video container per node, mapped 1-to-1, and the README says recording uses considerable CPU, roughly one CPU per video container, and is not supported for headless browsers (source: docker-selenium README). The contract: artifacts are written under the session id and collected before the sandbox is destroyed, so teardown waits for them.

perceptual-image-diff (no page yet) is the assertion that screenshots enable. Playwright uses pixelmatch with a threshold (default 0.2, from 0 strict to 1 lax) for what counts as a different pixel and an optional maxDiffPixels budget. It also warns that screenshots differ between browsers and platforms because of rendering and fonts, so baselines are per browser and platform (source: Playwright docs, Snapshots). For the grid that means the image must be pinned: the same browser build, fonts and operating system on every node, or the diffs measure the farm.

State and operations

The session map and queue are small but critical. Primary-backup replication (or an external store with its own replication) keeps the grid from losing running sessions on failover; losing the map orphans browsers until their idle timeout. Point-in-time recovery matters only for the results database, not the live grid. Selenium nodes send a heartbeat every 60 seconds by default and the Distributor health-checks all nodes every 120 (source: Selenium docs). Process supervision restarts crashed node processes, and Graceful draining is how a node leaves: stop accepting sessions, let running tests finish up to a deadline, then exit, so a rolling upgrade never kills a test mid-run. hdr-histogram (no page yet) records the one latency that decides the user's experience, queue wait, as a distribution, because the mean hides the minute a shard waited for a browser.

The scheduler and proxy have rules that rare interleavings break (a lease expiring while a command is in flight, a node dying while the queue retries). Deterministic simulation testing runs them on virtual time with simulated nodes and a seeded scheduler so each failure replays; Coverage-guided fuzzing hits the parsers that take client-controlled input, namely the capability matcher and the HTTP and frame parsers.

Data and control flow

One test session:

1. The runner (a Playwright worker or a Selenium client) sends a new-session request with its desired capabilities over mutual TLS to the front door.
2. The request waits in the admission queue, up to the queue timeout.
3. The scheduler takes the first request that matches a free slot, takes a lease on it and asks the node to start a session, which is quick if a warm browser or context exists and seconds if a sandbox must be created.
4. The node returns the session id and endpoint; the scheduler records id → node in the session map and answers the client through the front door.
5. Commands flow client → front door → node → browser, events come back, over HTTP or a WebSocket pipe. Each command renews the lease; the grid does not look inside.
6. Video and logs are written while it runs.
7. On quit, idle timeout or client disconnect, the node closes the browser, collects artifacts, destroys the sandbox or resets the context, and frees the slot.
8. The session map entry is removed; if the runner retries a failed test, it starts again at step 1 in a fresh session. Playwright, for example, discards the worker process and browser after a failed test and continues in a new worker (source: Playwright docs, Retries).

Time goes to queue wait, to browser start when no warm slot exists, and to the page itself; only the first two belong to the grid.

One suite run:

1. CI splits the suite with sharding: Playwright's --shard=x/y gives each of y jobs one share of the tests, balanced per test with fullyParallel and per file without it (source: Playwright docs, Sharding).
2. Each shard job's workers open sessions on the grid as above.
3. Shard reports are merged into one result; failed tests are retried within the retry budget.

Scaling

Limits come in this order:

1. Cold starts. Queue wait is dominated by browser start-up under bursts, because suites start all at once. Lever: a warm pool, then sizing it from the observed arrival rate.
2. Slots per node. Memory per browser, not CPU alone, sets the real number; Selenium's default of one slot per CPU is a conservative start. Lever: measure browsers per node under a real suite and set the cap from that.
3. Node count and packing. Add nodes on queue depth; pack sessions so nodes can be removed again.
4. The front door. Every command passes through it; use an event-driven proxy and add replicas with a shared session map or hashing.
5. Artifacts. Video encoding competes with the browser for CPU. Lever: record only on retry or failure, or encode off the node.
6. The suite itself. Past a point shard balance and test duration matter more than more browsers.

Failure modes

Real implementations

The alternative: run a headless browser on the runner's own machine. It is cheapest and most reliable for a small suite; a grid pays off with many parallel browsers, many versions, other operating systems or a pool shared by teams.

Sources

• Selenium documentation: Grid components (Router, Distributor, Session Map, New Session Queue, Node, Event Bus) and CLI options (timeouts, heartbeat, Docker, Kubernetes, drain options). components
• Playwright documentation: Isolation, BrowserType, Sharding, Retries, Docker, Snapshots. playwright.dev
• Chrome DevTools Protocol viewer. chromedevtools.github.io
• W3C WebDriver BiDi, Editor's Draft. w3c.github.io/webdriver-bidi
• W3C WebDriver Level 2. w3.org/TR/webdriver2
• docker-selenium README (video container, Dynamic Grid, slot selectors). github.com/SeleniumHQ/docker-selenium
• Selenoid, Zalenium and Moon READMEs and docs (status notes). Selenoid