← Back
Home › Systems › Frontier AI research

automated-ai-research

Lets language-model agents run the research loop of reading, proposing, running and judging machine-learning experiments in place of a human researcher.

AI Scientist, lab research agents

Open the automated-ai-research pageautomated-ai-research Agent loopagentic-ai: Agent memoryagentic-ai: Agent sandboxingagentic-ai: Citation groundingagentic-ai: Attaching answer spans to retrieved source passages and verifying each claim is supportedCode context retrievalagentic-ai: Context window managementagentic-ai: Keep an agent's growing history inside the model's window by truncating, clearing and summarizing old turns in a cheap-to-lossy ladder, at turn boundaries.›Edit applicationagentic-ai: Apply model-proposed code edits robustly: search/replace blocks, unified diffs, fuzzy anchor matching and conflict reportingSubagent orchestrationagentic-ai: Task planningagentic-ai: Tool callingagentic-ai: The model emits a structured call (id, name, JSON arguments) instead of prose; your code validates, runs it and returns a result tagged with the same id.›GPU cloudcloud: Where to rent GPUs for training and inference: hyperscalers for integration, neoclouds for price, spot and marketplaces for cheap batch work; interconnect decides multi-node training.›Durable workflow executiondistributed: Ordinary code made crash-proof by logging every side-effect result and replaying the code from that log; activities run at least once, workflow logic must be deterministic.›Gang schedulingdistributed: All-or-nothing co-scheduling of a job's tasksLLM-as-judgemachine-learning: Grade open-ended outputs with a language model under a rubric, then correct its known biases by order swapping, voting and calibration against human labels.›Experiment trackingml-training: Logging params, metrics, artifacts and code versions per run with an append-only metric store and comparisonPopulation based trainingml-training: Parallel training population with periodic exploit (copy best weights) and explore (perturb hyperparameters)Scaling lawsml-training: Loss vs parameters, data and compute power laws; compute-optimal model/data sizingSuccessive halving hyperbandml-training: Early stopping of weak trials by successive halving; Hyperband and ASHA schedulingFailure bisectionsearch-sort: Binary search over commits, batches or inputs to isolate a failureHybrid search and rank fusionsearch-sort: Merge a keyword (BM25) result list and a vector (ANN) list into one ranking, by ranks (RRF) or by normalized scores (convex combination).›Hermetic executionsystems: Run actions with only declared inputs in a sandbox and fixed environmentExponential backoff with jittersystems: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›Memory consolidationagentic-ai: Periodically merge near-duplicate memories, resolve contradictions, decay the unused and cap the store, so an agent's long-term memory stays small, current and cheap to search.›HNSWsearch-sort: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.›Egress proxy filteringsecurity: Forcing sandbox network traffic through a proxy with domain allowlists, TLS SNI checks and request loggingMicrovm isolationsecurity: Minimal KVM-based VMs (Firecracker-style) with tiny device model, fast boot and snapshot restoreProcess sandboxingsecurity: Run risky work in a separate, deliberately powerless process and let a small privileged broker do the few things it needs, over a narrow pipe.›Seccomp bpfsecurity: Syscall filtering with seccomp BPF programsEmbeddingsmachine-learning: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Text chunkingmachine-learning: Cut documents into token-bounded pieces along structure boundaries, keeping offsets and section paths, so each piece embeds well and can be cited.›Text classificationmachine-learning: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Cross-encoder rerankingsearch-sort: Re-score the top-k candidates of a cheap first stage with a model that reads query and candidate together; more accurate than separate embeddings, but one model call per pair.›Code symbol indexingcompilers: Extract definitions and references per file, give each symbol a stable string id, and store symbol to occurrences, re-indexing only files whose content hash changed.›+ 36 moreNear-duplicate detection withMinHash LSHmachine-learning: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Time decay scoringsearch-sort: Scoring items by recency with exponential decay or half-life, combined with relevanceEntity resolutionstorage: Decide which records describe the same real-world thing: block to cut the pair space, score pairs with a probabilistic model, cluster the matches transitively.›d-ary heapdata-structures: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.›SIMD programminghardware-os: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.›Scalar quantizationsearch-sort: Per-dimension int8/binary quantization of vectors with rescoringDNS message format and clientlookupsnetworking: Binary query/response format of DNS (RFC 1035) and how a stub client uses it: record types, name compression, UDP/TCP/EDNS sizes, TTL caching.›Http connect tunnelingnetworking: HTTP CONNECT tunnels for proxying TLS and arbitrary TCPPublic Suffix Listnetworking: Find the registrable domain (eTLD+1) of a host name by matching the longest rule in a community-maintained list of shared suffixes.›Glob matchingsearch-sort: Glob/gitignore pattern matching without backtracking blowupHardware virtualization kvmhardware-os: VT-x/AMD-V via KVM: vCPU run loop, VM exits, MMIO emulationNested paging epthardware-os: EPT/NPT two-level address translation for guests+ 45 moreSwiss tabledata-structures: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.›Union-findgraphs: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.›Record linkagesearch-sort: Probabilistic record matching (Fellegi-Sunter) with blocking for patient or entity indexes+ 17 more+ 2 more

How it is built

Architecture overview

An automated ML research system lets language-model agents do the loop a human researcher does: read the literature, propose a hypothesis, change code, run an experiment on GPUs, look at the numbers, decide what to try next, and write up what held. The agent part is the cheap part. The system around it is what turns an agent that writes plausible code into one that produces trustworthy results: a sandbox that contains it, a scheduler that gives it GPUs, a tracker that records exactly what ran, an experiment manager that spends the compute budget on promising branches, and an evaluator that does not trust the agent's own report. The one idea that makes it work is that every claim must trace back to a recorded run; an agent's text is a proposal, a logged metric from a hermetic run is evidence.

Published systems differ in how much of this they have. The AI Scientist (v1) ran a fixed pipeline from idea to LaTeX paper at a reported cost under $15 per paper. Its v2 dropped the human-written code templates and added an experiment manager agent steering a tree search. AIDE treats ML engineering as a tree search over code. Benchmarks such as MLE-bench, RE-Bench and PaperBench measure the result (see Sources).

Parts:

 literature index ─► idea generator ─► novelty check ─► plan (task DAG) ─┐
        ▲                  ▲                                            ▼
   agent memory ◄──── results + notes                       experiment manager (tree / halving / PBT)
        ▲                  ▲                                  │ proposes run configs, kills weak runs
        │                  │                                  ▼
        │            tracker (metrics, code hash, seeds) ◄── code workbench (edit, diff, test) ──► sandbox
        │                  ▲                                                                       │ job spec
        │                  └──────────── metrics, logs, checkpoints ◄── scheduler ◄─────────────────┘
        │                                                               (gang scheduling, GPU cloud)
        └──── writer (grounded citations) ◄── judge / reviewer ◄── only logged results are citable

How the parts fit together

Agent core

The core is a loop: the model emits a structured tool call, the harness executes it, and the result goes back into the context. Calls must parse, so arguments are checked against a schema (JSON Schema validation) and, where the model server supports it, generation is forced to match (Constrained decoding). Tools reach the workbench and scheduler over a JSON-RPC style protocol (json-rpc, no page yet). A research run lasts hours to days, so the context fills up long before the work is done: context window management summarises and prunes old steps, and prompt caching keeps the stable prefix (system prompt, task description, tool schemas) cheap to resend. Planning splits the campaign into tasks with dependencies (task-planning, no page yet); subagent-orchestration (no page yet) gives literature search, coding and analysis to separate agents whose summaries the planner holds. The loop itself (agent-loop, no page yet) is thin.

Knowledge: literature and memory

Idea generation needs the literature, and novelty checks need it more. Search is hybrid: a keyword index (Inverted index with BM25) catches exact method names, a vector index (HNSW over embeddings) catches paraphrases, and rank fusion merges the two lists. A cross-encoder re-scores the top candidates. Long papers are cut by chunking along sections so a passage keeps its context.

The writer must not invent support. citation-grounding (no page yet) attaches each claim to a source passage and verifies that the passage supports it, with text classification (entailment-style labels) and the reranker as checkers. Duplicate and variant papers collapse with MinHash and entity resolution.

Memory (agent-memory, no page yet) stores what the system itself has tried: hypotheses, outcomes, dead ends, in a vector index with time-decay-scoring (no page yet) so recent notes rank higher. Memory consolidation merges near-duplicates so the agent does not rediscover the same failed idea. Every note references run ids in the tracker, so memory stays tied to evidence.

Code workbench

The agent edits real repositories, so it needs the tools of a code editor. code-context-retrieval (no page yet) finds the code worth showing: symbol indexes (definitions and references from name resolution), a trigram-index (no page yet) for fast substring and regex search, and an inverted index underneath both. Because files change with every edit, parsing is incremental over lossless-syntax-trees (no page yet) that keep comments and whitespace, and a language server gives diagnostics and go-to-definition over one JSON-RPC conversation.

Applying a model's edit is where agents break files. edit-application (no page yet) accepts search/replace blocks or diffs and locates the target with fuzzy anchoring (fuzzy-string-matching and edit-distance, no page yet) when the model's quote is slightly off, then reports a precise error if it cannot. Myers diff shows the change back to the agent and to the reviewer, and three-way merge combines parallel branches of the same code, which tree search produces. When a run regresses, failure-bisection (no page yet) searches the sequence of changes for the commit that broke it.

Sandbox

Model-written code runs on machines with data and GPUs, and the model reads web pages and papers that may contain hostile instructions (Prompt injection defense). The sandbox (agent-sandboxing, no page yet) therefore has layers. Process sandboxing is the baseline: a separate process with namespaces, cgroup limits and a syscall filter, least privilege by default. Stronger isolation uses microvm-isolation (no page yet): a minimal KVM virtual machine with a tiny device model, as in Firecracker, so a kernel exploit by the agent's code does not reach the host. Network access is closed by default and opened through an egress-proxy-filtering (no page yet) allow-list of domains, which also logs what the agent fetched. Execution is hermetic (hermetic-execution, no page yet): declared inputs, pinned environment, so the same code and seed reproduce the same run. Irreversible actions wait for human approval.

Compute scheduler

An experiment is a job that needs N GPUs together or none: a distributed training run that gets half its GPUs deadlocks. gang-scheduling (no page yet) starts all tasks or none, placed by topology-aware-placement (no page yet) so that GPUs of one job share a fast interconnect, with bin packing to avoid fragmenting the cluster. priority-and-preemption (no page yet) lets urgent runs evict low-priority sweeps; a preempted job receives a termination signal and drains gracefully, writing a checkpoint (distributed-checkpointing, no page yet) so it can resume. Jobs are entries in a durable job queue with leases, retries and a dead-letter path, and each worker holds a lease with a fencing token, so a presumed-dead worker cannot write results over its replacement. Capacity comes from rented GPUs or an owned cluster; resource-usage-accounting (no page yet) attributes GPU-hours to each run for the manager's budget.

Experiment manager

The manager turns a fixed compute budget into the best decisions. Three tools, in the order of how much they assume:

  • Early stopping of weak runs. successive-halving-hyperband (no page yet) starts many configurations on a small budget, keeps the best fraction and gives them more, repeating; ASHA runs it asynchronously so workers never wait for a rung. It is a bandit allocation of compute, with top-k selection picking survivors.
  • Population methods. population-based-training (no page yet) trains a population in parallel and periodically copies the weights of good members to bad ones with perturbed hyperparameters.
  • Tree search over solutions. AIDE and the AI Scientist-v2 treat each candidate program or experiment as a node, expand promising nodes with an LLM and keep a frontier; this is best-first search in the family of MCTS, with the experiment result as the value.

Before spending a large run, scaling-laws (no page yet) extrapolate the outcome from small ones; the fit is a nonlinear least-squares problem (levenberg-marquardt, no page yet) and tells the manager whether a small-scale trend is worth scaling.

Tracking and provenance

Every run logs parameters, a hash of the code and data, seeds, metrics over time and artifacts (experiment-tracking, no page yet). The metric stream is an append-only segmented log; artifacts (checkpoints, plots, outputs) go to object storage addressed by hash (Content-addressed storage), so identical files are stored once and a result can be traced to the exact bytes. A merkle-dag (no page yet) links a run to its parent runs and code commits, the same structure as git. Long metric curves are downsampled for the dashboard with lttb-downsampling (no page yet), which keeps their shape.

Workflow engine

A campaign is a graph of steps (read, propose, implement, run, analyse, write) that may be restarted many times. Durable workflow execution logs each step's result, so after a crash the campaign resumes without redoing completed steps or repeating side effects; record-replay (no page yet) of model and tool outputs makes a failed campaign reproducible for debugging. Steps are ordered by topological sort of the dependency graph, and each side-effecting call carries an idempotency key. The event history is event sourcing in effect: the audit log of what the system did and why.

Evaluation

The evaluator is independent of the agent. A judge model grades ideas, results and drafts against a rubric; judges are noisy and biased, so they are validated against human labels (annotator-agreement, no page yet) and their scores calibrated. To rank many candidates, pairwise comparison with a bradley-terry-model (no page yet) is more stable than absolute scores, and it is the idea behind tournament-style ranking of hypotheses. Task-level success is measured with pass-at-k (no page yet). PaperBench uses such a judge against rubrics co-developed with paper authors; MLE-bench and RE-Bench score against leaderboards and human expert baselines.

Data and control flow

One idea, end to end:

  1. The idea generator reads the literature index and memory, proposes N hypotheses, and the novelty check searches hybrid retrieval for prior work; near duplicates are dropped.
  2. The judge ranks the survivors; the planner turns the top one into a task DAG: baseline, change, ablation, analysis.
  3. The code workbench retrieves the relevant files, the agent writes edits, applies them (fuzzy anchor, diff shown), and runs tests in the sandbox. A small smoke run (minutes) checks that the code trains at all.
  4. The experiment manager submits runs to the scheduler as a gang of GPUs with a time limit and priority. Rungs of successive halving or branches of the tree decide which to continue.
  5. Workers run in the sandbox, stream metrics to the tracker, and checkpoint on schedule. A preempted run is drained and resumed.
  6. The analysis step reads tracker data, not the agent's memory of the logs: plots and tables come from logged metrics, with seed variance. Surviving effects are replicated with fresh seeds before they count.
  7. The writer drafts with grounded citations and run ids for every number; the reviewer checks both, and an unsupported sentence is cut. Time goes mostly to step 5.

A failure and recovery: a worker disappears. Its lease expires, the queue reassigns the job with a new fencing token, and the new worker restores the last checkpoint. The workflow engine sees the step as still running and starts no duplicate.

Scaling

Limits in the order they bite:

  1. Evidence quality, not throughput. Many runs on many ideas produce false positives by selection. Levers: replication of winners, held-out evaluation the agent cannot see, pre-registered metrics.
  2. GPU supply. Experiments queue for GPUs. Levers: early stopping, small-scale proxies and scaling-law extrapolation, preemptible capacity with checkpointing, packing small jobs onto one GPU.
  3. Agent context and cost. Long campaigns exhaust the window. Levers: memory, prompt caching, narrow subagents, cheaper models.
  4. Scheduler fragmentation. Gang jobs wait for contiguous GPUs. Levers: bin-packing, backfill, priority classes.
  5. Human attention. Output exceeds what people can verify. Levers: calibrated automated review, a human gate on claims that matter, audit sampling.

Failure modes

FailureSymptomCauseFix
Fabricated resultspaper numbers not in any logthe model writes results from memory or expectationnumbers inserted from tracker queries only; a check that every figure resolves to a run id
Evaluation tamperingmetrics improve by changing the evalagent can edit the scoring code, tests or dataread-only mounts for evaluators, hidden held-out set, hash of eval code in the tracker
Selection inflationwinners do not replicatebest of many noisy runs reportedseeds, replication with fresh seeds, correction for number of trials
Sandbox escape or exfiltrationunexpected network calls or host accessuntrusted code or injected instructions from fetched pagesmicroVM isolation, deny-by-default egress, no secrets in the sandbox, approval for irreversible actions
Gang deadlockcluster idle yet jobs pendingpartial allocations holding GPUsall-or-nothing allocation, backfill
Edit applied to the wrong placetests fail in odd filesfuzzy match too looserequire unique match, show the diff, run tests, revert on failure
Context rotagent repeats failed attemptshistory pruned without notesmemory entries with run ids, consolidation, a "tried and failed" list in the prompt
Judge biashigh scores for fluent but weak workLLM judge prefers style, lengthrubrics, calibration against human labels, pairwise comparison with position swaps

Real implementations

namewhat it isrepolicencenotes
The AI Scientistend-to-end pipeline from idea to paper with an automated reviewerSakanaAI/AI-Scientist—v1; uses human-written code templates; read the repo licence before reuse
The AI Scientist-v2template-free version with agentic tree search and an experiment managerSakanaAI/AI-Scientist-v2—read the repo licence before reuse
AIDEML engineering agent framed as tree search over codeWecoAI/aidemlMITscaffold used in MLE-bench results
MLE-bench75 Kaggle competitions as an agent benchmarkopenai/mle-bench—grading against human leaderboards
PaperBenchreplicate ICML 2024 papers, graded by rubrics and an LLM judgeopenai/frontier-evalsMITrepository formerly named preparedness
SWE-agentagent with an agent-computer interface for code tasksSWE-agent/SWE-agentMITcode workbench reference
OpenHandsopen platform for software-development agents with sandboxed runtimeOpenHands/OpenHandsMITagent loop and sandbox in one platform
Ray (Tune)distributed execution with hyperparameter search schedulersray-project/rayApache-2.0ASHA, PBT and gang placement groups
MLflowexperiment tracking and artifact storemlflow/mlflowApache-2.0tracking server and model registry
Firecrackerminimal KVM microVM monitor for strongly isolated sandboxesfirecracker-microvm/firecrackerApache-2.0

Sources

  • C. Lu et al., The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, 2024. paper · code
  • Y. Yamada et al., The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search, 2025. paper · code
  • Z. Jiang et al., AIDE: AI-Driven Exploration in the Space of Code, 2025. paper
  • J. S. Chan et al., MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, 2024 (75 Kaggle competitions). paper
  • H. Wijk et al., RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts, 2024. paper
  • G. Starace et al., PaperBench: Evaluating AI's Ability to Replicate AI Research, 2025. paper
  • J. Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024. paper
  • L. Li et al., Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization, JMLR 2018. paper
  • L. Li et al., A System for Massively Parallel Hyperparameter Tuning (ASHA), MLSys 2020. paper
  • M. Jaderberg et al., Population Based Training of Neural Networks, 2017. paper
  • J. Kaplan et al., Scaling Laws for Neural Language Models, 2020. paper
  • A. Agache et al., Firecracker: Lightweight Virtualization for Serverless Applications, NSDI 2020. paper · code
  • J. Gottweis et al., Accelerating scientific discovery with Co-Scientist, 2025 (multi-agent hypothesis generation and ranking). paper

Similar systems

Share the most building blocks with this one.

ai-co-scientist 33 %coding-agent 27 %deep-research 25 %support-agent 18 %multi-agent-orchestrator 18 %