← Back
Home › Systems › Frontier AI research

rl-environments

Lets a language model agent practice a task many times in disposable environments and scores each attempt.

coding and computer-use gyms

Open the rl-environments pagerl-environments Accessibility treeagentic-ai: Extracting the browser/OS accessibility tree (roles, names, bounds) as a compact text view of a UI for agentsAgent tracingagentic-ai: Browser automation protocolagentic-ai: CDP / WebDriver BiDi commands and events over websockets to drive browsersInput event synthesisagentic-ai: Inject synthetic mouse, keyboard and touch events at OS or browser level (CGEvent, SendInput, uinput, CDP Input) with realistic timingModel context protocolagentic-ai: JSON-RPC protocol exposing tools, resources and prompts from servers to LLM clients, with capability negotiationScreen captureagentic-ai: Damage-region screen capture with GPU surfacesTool callingagentic-ai: The model emits a structured call (id, name, JSON arguments) instead of prose; your code validates, runs it and returns a result tagged with the same id.›User simulationagentic-ai: LLM-driven simulated users following scripted goals to test conversational agentsContainers and Kubernetescloud: Containers package a service with its dependencies; Kubernetes schedules them across machines. Use the smallest orchestrator that does the job.›Hdr histogramdata-structures: Log-linear bucketed histograms with bounded relative error for latency percentilesDeterministic simulationtestingdistributed: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Durable job queuedistributed: Persisted tasks that workers lease for a visibility timeout, ack on success and retry with backoff; at-least-once execution that survives worker and queue crashes.›Primary-backup replicationdistributed: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Record replaydistributed: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildProcess supervisionhardware-os: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Vm snapshot restorehardware-os: Snapshot an initialised VM/isolate heap and restore it for fast cold startsBenchmark contaminationdetectionmachine-learning: Detect test-set leakage into training data via n-gram overlap, perplexity and canary checksVerifiable rewardsml-training: Rewards from programmatic checkers (unit tests, math answer matching) for RL and data filteringCoverage-guided fuzzingsecurity: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Egress proxy filteringsecurity: Forcing sandbox network traffic through a proxy with domain allowlists, TLS SNI checks and request loggingMicrovm isolationsecurity: Minimal KVM-based VMs (Firecracker-style) with tiny device model, fast boot and snapshot restoreMutual TLSsecurity: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Process sandboxingsecurity: Run risky work in a separate, deliberately powerless process and let a small privileged broker do the few things it needs, over a narrow pipe.›Dataset versioningstorage: Immutable, content-hashed dataset snapshots with manifests and lineage so experiments are reproduciblePoint-in-time recoverystorage: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.›Autoscalingsystems: Target-tracking scaler with stabilization windows, cooldowns and hysteresisGraceful drainingsystems: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Hermetic executionsystems: Run actions with only declared inputs in a sandbox and fixed environmentWarm pool provisioningsystems: Keep pre-initialized instances ready to cut cold startsDistributed tracingsystems: Give each request a 128-bit id, record timed spans linked by parent ids, pass the ids in a header across every hop, and join the spans back into a tree afterwards.›Telemetry samplingsystems: Head, tail and dynamic sampling of traces, events and logs to a budgetServer-sent eventsnetworking: One long-lived HTTP response carries text events one way; each event may carry an id, so a reconnecting client names its last one and the server resumes.›OAuth 2 and OpenID Connectsecurity: Delegated authorization (OAuth 2) and login on top of it (OIDC); use authorization code + PKCE for every user-facing client.›Json rpcnetworking: JSON-RPC 2.0 requests, notifications and batchingJSON Schema validationsecurity: Compile a JSON Schema once, then check every untrusted document against it, within hard size and depth limits, and report failures as JSON Pointers.›Constrained decodingagentic-ai: Force LLM output to match a regex, JSON schema or grammar by masking, at every step, the tokens that would leave the language.›Dialogue state trackingmachine-learning: Keep a structured belief of the user's goal (slot → value, plus requests) and update it every turn from NLU output, the old state and the last system action.›Template-based text generationmachine-learning: Realise facts as sentences by filling slots in alternative templates, with agreement, referring expressions and seeded variation; every word traces to data or a template.›Token samplingml-training: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.›Cooperative schedulingconcurrency: Green threads/coroutines yielding at await points on a user-space schedulerFault injectiondistributed: Inject latency, errors, kills and partitions via proxies, netem or hooks+ 39 moreDistributed ID generationdistributed: Mint unique, roughly time-sorted IDs on many nodes with no per-ID coordination: Snowflake (time, worker, sequence) or UUIDv7 (time, random).›Protobuf wire formatnetworking: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolutionColumnar storagestorage: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Hash based samplingdata-structures: Deterministic sampling by hashing an entity id so all its events are kept or dropped togetherReservoir samplingdata-structures: Uniform k-sample from a stream of unknown length (Algorithm R/L)Token bucketsystems: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.›Csprngsecurity: OS-backed cryptographically secure random numbers (getrandom), and when a fast PRNG is not enoughJSON Web Token (JWT)security: A signed, URL-safe token carrying claims; the verifier's job is the checklist (pinned algorithm, key by kid, exp, aud, iss), not the format.›Byte-pair encoding (BPE)tokenizationmachine-learning: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.›Finite state transducersearch-sort: Minimal acyclic FST mapping sorted keys to values; compact term dictionaries with prefix/fuzzy searchStatechartsgraphics: State machines with nesting, parallel regions, history and entry/exit actions, so UI and animation logic stays linear in size instead of exploding.›Intent classification and slotfillingmachine-learning: Turn one utterance into an intent label and typed slot values with a joint model: a sentence-level classifier plus a token-level BIO tagger on a shared encoder.›+ 13 moreBit packing and PFORcompression: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›Delta encodingcompression: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Dictionary encodingcompression: Map values to small integer codes via a dictionary for compact, comparable columnsVectorized executionstorage: Query operators pass batches of about 1-2K values per column instead of single tuples, amortizing interpretation overhead and enabling tight SIMD loops.›Zone mapsstorage: Per-block min/max (and sparse primary) indexes to skip data during scansHierarchical state machinesystems: Nest states so a substate inherits its parent's reactions and only overrides what differs, with ordered entry and exit actions along the path.›Named entity recognitionmachine-learning: Find the character spans of entities in text and give each a type, by labelling every token and decoding a valid label sequence.›Text classificationmachine-learning: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›+ 8 moreVarint (LEB128)compression: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.›Convolutional codes and theViterbi decodersignal-processing: Streaming error-correcting code: each output bit is a parity of the last K input bits; the Viterbi decoder finds the most likely path through a 2^(K-1)-state trellis.›Transformer blockml-training: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›Calibrationmachine-learning: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.›Linear and logistic regressionmachine-learning: Fit a weighted sum of features to a number (least squares) or a probability (logistic); the baseline every model must beat.›+ 1 moreLoss functionsmachine-learning: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›L-BFGSnumerics: Quasi-Newton minimizer that approximates the inverse Hessian from the last m gradient differences via the two-loop recursion; O(m·n) memory and work per step.›LU and Cholesky factorizationnumerics: Factor A = PLU (any square matrix, partial pivoting) or A = LLᵀ (SPD) once in O(n³), then solve each right-hand side in O(n²).›

How it is built

Architecture overview

An RL environment platform is the part of a model-training stack that lets a language-model agent practise a task thousands of times and tells it how it did. For a coding agent the task is "fix this issue in this repository"; for a computer-use agent it is "book this ticket in this web app". Each attempt (a rollout) happens inside a disposable sandbox that holds the task's world: a checked-out repository, a desktop, a website. The agent acts on it through tools, and when it stops, a verifier looks at the final state and produces a reward. The trainer collects many rollouts, turns rewards into a learning signal, and updates the model.

The idea that makes it work is that the environment is a function from a task spec to a scored trajectory, with isolation and reproducibility as hard requirements. Isolation, because the agent executes arbitrary code and may try anything that raises its reward; reproducibility, because a reward is only a training signal if the same behaviour gets the same score. Everything else here serves those two properties at the scale of many thousands of concurrent sandboxes. The interface (reset, step, reward) comes from Gym (Brockman et al., 2016); a step is now a tool call into a whole operating system.

Parts:

 task set (versioned, hashed) ──► rollout scheduler (durable queue, leases) ◄── trainer (policy server + RL update)
                                       │ lease                                     ▲  rewards, trajectories
                                       ▼                                           │
                     sandbox pool ──► sandbox (microVM / container, from snapshot)
                     (warm, autoscaled)  ┌──────────────────────────────────────┐
                                         │ task world: repo / desktop / website │
                       model server ◄───►│ tool server (MCP, shell, browser)    │──► egress proxy ──► allowed hosts only
                      (tool calls)       │ simulated user / services            │
                                         └─────────────┬────────────────────────┘
                                                       │ final state
                                                       ▼
                                                  verifier ──reward──► trajectory store ──► tracing, replay, audits

The trainer owns the learning algorithm (for example Policy gradient and PPO, or the group-relative variant introduced for DeepSeekMath, Shao et al., 2024); the platform owns what is true about a rollout: what the agent saw, did and earned.

How the parts fit together

Task set

A task is a package: a starting state (an image or filesystem snapshot plus setup commands), an instruction, and a checker. SWE-bench (Jimenez et al., 2023) set the pattern for code: a real repository at a commit, an issue as the instruction, and the repository's tests as checker (tests that must go from failing to passing, and tests that must keep passing). OSWorld (Xie et al., 2024) and WebArena (Zhou et al., 2023) do the same for desktop and web use, with a per-task evaluation script.

Tasks are stored by content: file system layers are blobs under their hash (Content-addressed storage, Cryptographic hashes) in Object storage, and the task set is a manifest of hashes plus metadata. That gives dataset-versioning (no page yet): a manifest hash always means the same bytes, and a run records which manifest it used. Training and held-out tasks must not leak into each other: overlap checks (benchmark-contamination-detection, no page yet, built on Near-duplicate detection (MinHash) and a Bloom filter) run on every new batch of tasks.

Sandbox runtime

The agent runs code it wrote, so the sandbox is the security boundary. There is a ladder of isolation, from cheap to strong. A process sandbox (Process sandboxing, Least privilege and isolation) uses Linux cgroups and namespaces with a syscall filter: fast, shares the host kernel, fine for trusted-ish code. A container (Containers and Kubernetes) packages the task's files and dependencies with the same kernel sharing. A user-space kernel such as gVisor intercepts system calls so the host kernel is not exposed directly. A microVM (microvm-isolation, no page yet; Firecracker, Agache et al., 2020) gives each rollout its own guest kernel on a minimal KVM device model, which is the right level when the agent can run anything, including a root shell and a browser. The choice is a trade-off between boot time and memory overhead on one side and blast radius on the other.

Fast start is where the platform wins or loses. The environment is initialised once and snapshotted: vm-snapshot-restore (no page yet) saves guest memory and device state and restores it per rollout, mapping the memory file lazily (mmap and the page cache) so only touched pages are read. Disk state is layered: a read-only base image shared by all rollouts under a small writable layer (overlay filesystem, no page yet) that is discarded when the rollout ends. Hermetic execution (hermetic-execution, no page yet) completes it: the checker runs with only declared inputs in a fixed environment.

Sandbox pool

Rollouts arrive in bursts (a training step wants thousands at once) and idle sandboxes are expensive. The pool keeps a warm pool of restored but unassigned sandboxes (warm-pool-provisioning, no page yet), sized by an autoscaler (autoscaling, no page yet): Sliding-window counters measure the arrival rate, Exponential smoothing forecasts it, and a PID control-style loop with a cooldown moves the size to the target without oscillating. Health checking removes sandboxes whose daemon does not answer. Contract with the scheduler: a sandbox is handed out once, used by one rollout, and destroyed; reuse would leak state and corrupt rewards.

Tool interface

The agent sees the world through tools. A coding agent gets a shell, file edit and search; a computer-use agent gets screenshots, the accessibility tree of the UI as compact text (accessibility-tree, no page yet), synthetic mouse and keyboard input (screen-capture and input-event-synthesis, no pages yet), or a browser driven by a debugging protocol (browser-automation-protocol, no page yet). The model emits Tool calling calls (name and JSON arguments) that the tool server validates against a schema (JSON Schema validation) before executing, and Constrained decoding can force well-formed calls during sampling. SWE-agent (Yang et al., 2024) showed that the design of this interface, the agent-computer interface, changes the results as much as many model changes.

The Model Context Protocol (model-context-protocol, no page yet) is a common wire format: JSON-RPC between a client and tool servers, over a stream or Server-sent events. Tools that need an identity (a mock mail account) get per-rollout tokens (OAuth2 and OIDC, JWT) with a short lifetime and only the task's scopes.

Network boundary

By default a sandbox has no network. Tasks that need it get it through an egress proxy (egress-proxy-filtering, no page yet): all traffic passes one proxy with a domain allowlist, which needs the registrable domain of a host (Public suffix list) so that allowed.example.com does not permit allowed.example.com.attacker.net. DNS is controlled too (DNS protocol): it is a path out. Mutual TLS authenticates the sandbox to platform services. It also protects the evaluation: an agent that can fetch the upstream fix is measured on lookup.

Verifier and reward

Reward comes from a checker, ideally one that is cheap, deterministic and hard to fool. Programmatic checkers are called verifiable rewards (verifiable-rewards, no page yet; named and used for post-training in Tulu 3 as RLVR, Lambert et al., 2024): unit tests for code, exact or normalised answer match for maths (Text normalization), a database state comparison for a web task. A checker that runs the agent's code must itself run in the same sandbox discipline (hermetic-execution, Process sandboxing). Where no programmatic check exists, an LLM as judge scores the end state or the trajectory; it is more flexible and more gameable, so it is used with a rubric and often combined with a programmatic part.

Sampling many attempts is scored with pass@k (pass@k, no page yet): with n samples of which c are correct, an unbiased estimator of the chance that at least one of k succeeds (Chen et al., 2021).

Reward hacking is the central risk: verifiers are code, and the agent edits the tests, reads the answer file, or exploits a checker quirk. Defences are structural: tests the agent cannot see or write, a checker outside the sandbox, and adversarial testing of the checker with Coverage-guided fuzzing and Code coverage instrumentation, which show which verifier branches the tests exercise and which inputs break its assumptions.

Simulated users and services

A simulated user (user-simulation, no page yet) is an LLM given a scripted goal and persona that answers the agent's questions; the scenario state is tracked (Dialogue state tracking) so the checker can compare the final state with the goal. Cheap deterministic parts use Template text generation; stochastic parts use fixed seeds and Sampling strategies with low temperature. Mock services (a shop, a mail server) are small state machines (Statecharts) whose state is part of the snapshot.

Rollout scheduler

Rollouts are long (minutes to hours), fail for many reasons (agent crash, sandbox host loss, model server timeout), and must be neither lost nor double-counted. The scheduler is a Durable job queue: a rollout request is persisted, a worker takes a lease (Leases) and heartbeats; if the worker disappears, the lease expires and the rollout goes back to the queue. A Fencing token on writes to the trajectory store prevents a zombie worker, whose lease expired but which keeps running, from overwriting the new attempt's result. Retries use Exponential backoff with jitter; rollouts that keep failing go to a Dead-letter queue for inspection (a broken task, not a broken agent). Completion messages carry an Idempotency key so a duplicate delivery does not count a reward twice. Deployments use Graceful draining, and Process supervision keeps the sandbox daemon and tool servers alive.

Trajectory store and observability

A trajectory is the complete record: prompts, tool calls, observations, timings, token logprobs, reward. It is appended to a Write-ahead log, then compacted into Columnar storage for training and analysis, with Point-in-time recovery (and Primary-backup replication if losing it costs weeks of compute). Each rollout carries a trace (Distributed tracing; telemetry sampling, no page yet, keeps volume sane while always keeping failed or very long rollouts), so a slow step can be placed: model, tool, sandbox start or verifier. agent-tracing (no page yet) gives the trace its shape for an agent: one span per model call, tool call, sandbox restore and verifier run, nested under the rollout and carrying the tool arguments, the observation and the token counts, so the same record serves debugging, cost accounting and the search for reward hacking. The timings in those spans feed hdr-histogram (no page yet), which keeps the latency percentiles of the steps that set rollout throughput: sandbox restore time, tool-call round trip, model call time and verifier time. A mean hides the slow restore or the single long tool call that stalls a whole training batch; a p99 per step shows it, and histograms from many workers merge without losing the tail. Traces are also how reward hacking is found: sort by high reward and low effort, and read.

Determinism and fault testing

Where the environment is deterministic (seeded PRNG (PCG, xoshiro), fixed clock, no external network) a rollout can be replayed from its recorded model outputs and tool results (record-replay, no page yet) to reproduce a bug or an exploit. Agents must also cope with failing tools, so the platform injects faults (fault injection, no page yet): latency, errors, killed processes. Deterministic simulation testing applies to the platform itself: run scheduler, queue and pool in one thread under a seeded clock and fault generator, and every failure becomes a replayable seed.

Data and control flow

One rollout:

  1. The trainer asks the scheduler for N rollouts of tasks drawn from the current task manifest; each request is persisted.
  2. A worker leases a request and takes a sandbox from the warm pool (restored from the task's snapshot), with network policy, tool servers and tokens attached.
  3. The harness sends the instruction to the model server. The model emits a tool call; the tool server validates and runs it in the sandbox and returns the observation. Repeat until the agent stops, a step budget or a time budget runs out.
  4. The verifier inspects the final state (outside the agent's reach) and computes the reward, possibly in several components (tests passed, format, cost).
  5. The worker writes the trajectory and reward with its fencing token, acks the job, and destroys the sandbox.
  6. The trainer waits for enough rollouts, updates the model, and publishes the weights to the model servers. Wall time is mostly generation and tool execution; start-up should be a minor share, which is what snapshots and the warm pool are for.

Scaling

In the order you meet them:

  1. Sandbox start latency and density. Many thousands of short- to medium-lived sandboxes. Levers: snapshots with lazy memory loading, shared read-only base layers, overcommit with limits, a warm pool sized from the arrival forecast.
  2. Host capacity. CPU, memory and disk I/O of the sandbox hosts rather than GPUs, often the dominant cost for agentic tasks. Levers: pack by measured use, cap runaway processes with cgroups, put heavy tasks on separate pools.
  3. Model-server throughput. Rollouts wait on generation. Levers: Continuous batching on the model servers, asynchronous rollouts, truncating long episodes.
  4. Verification cost. Test suites can take minutes. Levers: run only relevant tests, cache by content hash.
  5. Data volume and task supply. Trajectories are large (columnar storage, sampled traces); good verified tasks are scarce (automated construction, difficulty filtering).

Failure modes

FailureSymptomCauseFix
Reward hackingreward rises but real quality does notagent edits tests, reads the oracle, exploits checker bugshidden tests, checker outside the sandbox, adversarial review, fuzz the verifier
Sandbox escapehost or other tenants affectedshared kernel, wide syscall surface, over-privileged mountmicroVM or user-space kernel for untrusted code, least privilege, no host mounts
Leaked answerspass rate jumps on a tasknetwork access to the upstream fix, solution files in the imageno network by default, allowlist proxy, scrub images
Flaky verifierthe same behaviour gets different rewardstime, randomness, order dependence, network in testshermetic execution, seeded RNG, retries only for infrastructure faults
State leak between rolloutsa rollout starts "pre-solved"sandbox reused, shared writable layerone-shot sandboxes, read-only base layers, destroy on exit
Lost or duplicated rolloutsmissing or double-counted samplesworker death, duplicate deliveryleases, idempotency keys, fencing tokens
Silent environment changescores shift between runstask images updated in placeimmutable, content-hashed task sets; log the manifest hash

Real implementations

namewhat it isrepolicencenotes
SWE-benchtask set and containerised evaluation harness for issue-fixingSWE-bench/SWE-benchMITthe coding-task pattern
SWE-agentagent with a designed agent-computer interfaceSWE-agent/SWE-agentMITthe tool interface seam
OSWorldreal-computer environments and evaluators for multimodal agentsxlang-ai/OSWorldApache-2.0desktop tasks in virtual machines
WebArenaself-hosted web environment and tasks for agentsweb-arena-x/webarenaApache-2.0web tasks with evaluators
verifierslibrary for building RL environments and verifiable rewards for LLMswillccbb/verifiersMITenvironment and reward abstractions for training
OpenEnvframework for isolated agent environments with a Gym-style APImeta-pytorch/OpenEnvBSD-3-Clauseenvironments served as containers
Firecrackerthe microVM monitor behind many multi-tenant sandboxesfirecracker-microvm/firecrackerApache-2.0KVM-based, snapshot support
gVisoruser-space application kernel for container sandboxinggoogle/gvisorApache-2.0the middle rung of the isolation ladder

Frontier labs' internal platforms are not public.

Sources

  • G. Brockman et al., OpenAI Gym, 2016. paper
  • C. Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023. paper
  • J. Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024. paper
  • T. Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, 2024. paper
  • S. Zhou et al., WebArena: A Realistic Web Environment for Building Autonomous Agents, 2023. paper
  • A. Agache et al., Firecracker: Lightweight Virtualization for Serverless Applications, NSDI 2020. paper
  • N. Lambert et al., Tulu 3: Pushing Frontiers in Open Language Model Post-Training, 2024 (RLVR). paper
  • Z. Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, 2024. paper
  • M. Chen et al., Evaluating Large Language Models Trained on Code, 2021 (pass@k estimator). paper
  • Model Context Protocol: modelcontextprotocol.io

Similar systems

Share the most building blocks with this one.

agent-browser-isolation 46 %browser-test-grid 32 %computer-use-agent 31 %serverless 30 %ci-runner-fleet 28 %