ai-co-scientist

How it is built

Architecture overview

An AI co-scientist turns a research goal written in plain language into a ranked list of testable hypotheses and research proposals, and hands the best ones to people who run the experiments. The model never works alone: a supervisor keeps a task queue, a pool of workers executes specialised agent roles (generate, review, rank, cluster, evolve, summarise), and a persistent state records every hypothesis, review and tournament result so the run can be restarted and so later rounds learn from earlier ones. The one idea that makes it work is generate, debate, evolve: hypotheses are compared pairwise in a tournament whose Elo scores decide which get refined, and review feedback is not trained into the model but appended to the agents' prompts in the next round. More compute means more rounds of the loop, not a bigger model. Google's AI co-scientist (built on Gemini 2.0; Gottweis et al. 2025) is the reference design; Sakana's AI Scientist and FutureHouse's Robin use the same ingredients but differ in what follows the hypothesis: write and run code, or plan wet-lab assays.

Parts:

• Goal parser and plan configuration — turns the research goal into criteria (novelty, feasibility, constraints) the other parts read — Constrained decoding, JSON Schema validation
• Supervisor and task queue — decides which agent action runs next and how much compute each role gets — Durable job queue, Durable workflow execution, Multi-armed bandits, Leases, task-planning (no page yet), agent-loop (no page yet), subagent-orchestration (no page yet)
• Context memory — persistent store of agent and system state, statistics and feedback — agent-memory (no page yet), Event sourcing, Write-ahead log
• Generation — literature search and simulated debate produce new hypotheses — Tool calling, Hybrid search and rank fusion, PDF text extraction, Text chunking, citation-grounding (no page yet), relation-extraction (no page yet), Entity linking
• Reflection — peer-review style critique in several depths — LLM-as-judge, Cross-encoder reranking, citation-grounding (no page yet), self-consistency-voting (no page yet)
• Ranking tournament — pairwise debates, Elo scores, match scheduling — Elo rating, LLM-as-judge, Top-k selection
• Proximity — similarity graph of hypotheses for clustering and de-duplication — embedding-inference (no page yet), Near-duplicate detection with MinHash LSH, DBSCAN, Union-find
• Evolution and meta-review — create improved hypotheses and distil review patterns into prompt feedback — Context window management, Prompt caching
• Scientist in the loop — goal edits, manual reviews, own hypotheses, choice of what to test — Human-in-the-loop approval, Tamper-evident audit log

 scientist ──research goal──► goal parser ──plan config──► SUPERVISOR ◄──── stats ────┐
     ▲   ▲                                                 │  weights, queue          │
     │   └── reviews, own hypotheses, redirects ───────────┼───────────────┐          │
     │                                                     ▼               │          │
     │              ┌──────────────── task queue ──────────────────┐       │          │
     │              ▼            ▼          ▼          ▼          ▼       │          │
     │         Generation   Reflection   Ranking   Proximity   Evolution  │          │
     │         (web search) (reviews)   (Elo debates) (graph)   (new, never edits)    │
     │              └────────────┴──── workers: LLM + tools ───┴──────────┘          │
     │                                    │ read / write                              │
     │                                    ▼                                           │
     │                         ┌── CONTEXT MEMORY ──┐   Meta-review: feedback ────────┘
     │                         │ hypotheses, reviews│   appended to next prompts
     │                         │ tournament, stats  │
     │                         └────────────────────┘
     └── research overview + ranked hypotheses ◄── Meta-review
                                     │
                          (outside the system) experiments in the lab

There are two time scales: inside the system, asynchronous model calls that need no person; outside it, a scientist picks hypotheses and runs real experiments, which is where the real validation happens.

How the parts fit together

Goal parser and plan configuration

The goal is natural language, possibly with long background documents. The system parses it into a research plan configuration: preferences (for example only novel hypotheses), attributes and the criteria used later for auto-evaluation, such as novelty and experimental feasibility (Gottweis et al. 2025, section 3.2). The contract is that this configuration is data every agent prompt is built from. In a build of your own, make it a typed record validated before the run (JSON Schema validation) and force structured output where a model writes it (Constrained decoding); a malformed configuration silently changes what "good" means for the whole tournament.

Supervisor, task queue and workers

The supervisor does not reason about science. It builds a task queue, assigns weights and resources to the agent roles, and periodically computes statistics (how many hypotheses are waiting for review, how the tournament is progressing, whether new ideas from Generation or improved ideas from Evolution win more matches) and writes them to memory. They drive the next round of sampling and the stop decision. Workers take one agent action off the queue, call the model and tools, and write the result back.

This is a Durable job queue with a scheduler on top. Each task is leased to one worker (Leases) so a crashed worker's task reappears, and retries use Exponential backoff with jitter because most failures are rate limits and timeouts of model APIs. Because every step can be repeated safely from stored state, the run behaves like Durable workflow execution without a full workflow engine; compare DAG workflow execution, which fixes the order of steps in advance; here the next task depends on tournament results. The paper says the supervisor weights and samples agents using the statistics, without naming an algorithm. Treating each role as an arm and its recent win rate as the reward (Multi-armed bandits) is our reading, not a claim of the paper.

task-planning (no page yet) is how the supervisor turns the goal and the current statistics into the next batch of tasks: for example, generate more hypotheses, review the unreviewed ones, or run matches between close pairs, with a dependency noted where one task needs another's result. Each worker runs an agent-loop (no page yet): call the model with the role's prompt and the task's context, execute the tool calls it asks for, feed results back, and stop when the role's output is complete or a budget of steps is spent. subagent-orchestration (no page yet) is the contract between the two: the supervisor starts a worker as a subagent with its own fresh context and only the task's inputs, and receives a compact result (a hypothesis, a review, a match verdict) that it writes to memory, instead of the worker's whole transcript.

Context memory

Memory holds the state of the whole run: every hypothesis with its reviews and Elo score, the proximity graph, the meta-review text, the research overview and the supervisor's statistics. The paper states two uses: long-horizon iteration, and restart after a failure of any component (section 3.2). That makes it the system's source of truth, so the useful shape is an append-only record of events with the current state derived from it (Event sourcing, Write-ahead log). The agents' own context is not memory: the model sees only what a task puts in its prompt, which is why Context window management decides what a review sees (the hypothesis, its reviews, the relevant papers) and what is left out. agent-memory (no page yet) covers recall across sessions; here the store is per research goal.

Generation

Generation produces the first list of hypotheses. The paper names four techniques: iterative literature search with web tools, simulated scientific debates between expert personas, finding testable intermediate assumptions and aggregating them into hypotheses, and "research expansion" into areas the current set does not cover, guided by the previous overview. The tool contract is Tool calling: a search call returns documents, which are cleaned with PDF text extraction, cut with Text chunking, and merged from keyword and vector hits with Hybrid search and rank fusion. The paper names search and retrieval as the primary tools, optionally over a private paper repository; the retrieval stack above is the standard way to build that, not a description of Google's internals. Grounding is only real if each claim points to the passage that supports it: citation-grounding (no page yet).

Literature is also read as structured claims. relation-extraction (no page yet) pulls (subject, relation, object) triples from the cleaned passages, for example that a compound inhibits a protein in a given cell type, and Entity linking maps each mention to one identifier, or to none when it is not in the knowledge base, so "TNF-alpha" and "tumor necrosis factor" count as one thing. The linked triples feed two uses: debate personas can be handed concrete, cited facts about the goal's entities, and a new hypothesis can be checked against what is already known, which supports the novelty criterion that Reflection scores.

Reflection

Reflection plays the peer reviewer, in several depths so that cheap filters run first (section 3.3.2). An initial review without external tools discards flawed or non-novel hypotheses quickly. A full review adds literature search. A deep verification review splits a hypothesis into assumptions and sub-assumptions and checks each alone, which finds a wrong step even when the hypothesis as a whole sounds plausible. Further types look at whether the hypothesis explains existing observations, simulate the proposed mechanism step by step, and adapt to what the tournament has shown. All of these are LLM-as-judge calls with a rubric. Such judges have known biases; the co-scientist's answer is to decompose the judgement, ground it in searched papers, and let a person add reviews. Cross-encoder reranking is the cheap way to pick which retrieved papers a full review reads.

Reviews are noisy, so a verdict that decides whether a hypothesis survives is not taken from one sample. self-consistency-voting (no page yet) draws several independent reviews, or deep-verification runs, at nonzero temperature and keeps the majority verdict per assumption; a hypothesis is cut only when most samples find the same flaw, and split votes are sent on with a note instead of being discarded. The same vote settles a tournament debate whose two sides are close, at the price of a few extra model calls on exactly the matches where it matters.

Ranking tournament

Ranking is the selection pressure of the whole system. New hypotheses enter at an Elo rating of 1200, and pairs are compared in a debate: top-ranked hypotheses in multi-turn debates, lower-ranked ones in single-turn comparisons. Matches are scheduled so that similar hypotheses meet (using the proximity graph) and newer and higher-ranked ones play more often (Gottweis et al. 2025, section 3.3.3). Elo rating is the update rule; Glicko-2 adds an uncertainty per rating, which would suit hypotheses with few matches, and bradley-terry-model (no page yet) is the batch fit of the same pairwise model. The paper notes that Elo has assumptions and limits but is a good proxy for relative ranking. Every match is model calls, so the scheduler decides where evidence is worth the money, and Top-k selection picks what goes to the evolution step.

Proximity

Proximity builds a graph of how similar the hypotheses are for the research goal. It serves clustering and de-duplication for the scientist and the choice of opponents in the tournament. The paper does not say how similarity is computed. A typical build embeds each hypothesis with embedding-inference (no page yet: a pretrained embedding model served as is, with batching and a vector cache, and no training in this system) and merges duplicates with Union-find or groups them with DBSCAN; for text that is near-verbatim, Near-duplicate detection with MinHash LSH is cheaper. Without it the tournament fills with paraphrases of one idea.

Evolution and meta-review

Evolution takes top-ranked hypotheses and makes new ones: grounding them with more literature, making them more feasible, combining several, simplifying them, or moving away from them on purpose. One design rule matters: it only adds hypotheses and never edits or replaces existing ones, so a flawed "improvement" cannot damage a good hypothesis; it has to win matches in the tournament like everything else.

Meta-review reads the tournament and the reviews, finds recurring points (the paper's example is a reviewer that catches a blood-brain-barrier problem in 90 % of reviews but should catch it in all) and writes a critique that is appended to the prompts of the other agents in the next round. This is learning without gradient steps: the prompt is the parameter, and it relies on long-context models. It also periodically writes a research overview that summarises the top hypotheses and suggests experiments, which can be formatted into a grant template with Constrained decoding. A long shared feedback prefix suits Prompt caching.

Scientist in the loop

The scientist can rewrite the goal, add reviews, enter their own hypotheses (they are ranked next to the system's), and steer the system toward given papers. All three wet-lab validations in the paper had experts select which hypotheses to test. A real deployment treats that as an approval flow (Human-in-the-loop approval). An append-only record of who changed what (Tamper-evident audit log) lets you explain why a hypothesis was promoted.

Data and control flow

One research goal, end to end (inside the system):

1. The scientist submits the goal; the parser writes the plan configuration to memory.
2. The supervisor seeds the queue: Generation tasks first (literature search, debates), each writing hypotheses into memory with a starting Elo of 1200.
3. Reflection tasks pick hypotheses without a review. Initial review is tool-free and cheap; survivors get full and deep-verification reviews that read papers fetched through the search tool.
4. Ranking tasks pick pairs (similar, new, high-ranked first), run the debate, and update both ratings.
5. Proximity tasks update the similarity graph; clusters and duplicates are marked.
6. Evolution tasks take the current top hypotheses and add new ones, which go back to step 3.
7. Meta-review summarises review and debate patterns into feedback text; the supervisor writes statistics to memory and reweights the roles. Time goes to model calls and web fetches; queue, Elo update and graph are small.
8. When the supervisor decides the run is done, Meta-review writes the overview and the ranked list goes to the scientist.

One hypothesis, outside the system:

1. The scientist reads the overview, adds reviews or changes the goal, and selects candidates.
2. The lab runs the experiment (days to weeks; this is the slowest step of the whole loop).
3. Results go back as feedback or as a new goal. The co-scientist paper reports validated cases as separate studies; Robin (below) closes this loop with a data-analysis agent.

Scaling

In the order they bite:

1. Model-call throughput and cost. Every review and every match is a long-context call. Levers: cheap tool-free filters before expensive reviews, single-turn matches for low-ranked pairs (the paper's own optimisation), Prompt caching for the shared goal and feedback prefix, more workers.
2. Tournament size. All-pairs comparison grows quadratically. Levers: schedule by proximity and rank, drop low performers, de-duplicate first.
3. Context length. Papers, reviews and feedback compete for the window. Levers: retrieve passages, summarise reviews, keep meta-review feedback short.
4. Search quality. The system reads only open-access literature and has little access to negative results (section 5), so more compute eventually does not help. Levers: private repositories, domain databases, specialised models as tools.
5. Evaluation validity. The auto-evaluation Elo is not independent ground truth. The paper reports that the best and top-10 Elo rise over time across 203 research goals and that higher Elo goes with higher accuracy on GPQA diamond questions (78.4 % top-1 when taking the highest-rated answer), but it states the metric can favour attributes scientists do not care about. Lever: expert review and wet-lab tests, which cost the most and scale the least.

Failure modes

Real implementations

The alternative: a single "deep research" agent that searches, reads and writes one report. It is cheaper and good for synthesis, but nothing compares candidates or learns from review patterns. Pick the tournament when the output is a ranked set of ideas that are expensive to test; pick one research agent for a grounded summary. If validation is a computation rather than a lab assay, the AI Scientist family uses experiment results instead of a tournament as the signal.

Sources

• J. Gottweis et al., Towards an AI co-scientist, 2025 (v1 title; later versions are titled Accelerating scientific discovery with Co-Scientist). paper (agent roles, Elo 1200, tournament, GPQA concordance, scaling plots, limitations: sections 3 to 5).
• Google Research blog, Accelerating scientific breakthroughs with an AI co-scientist, 19 February 2025. blog
• C. Lu et al., The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery, 2024. paper (code-sandbox failures and the cost per paper are in section 8 and the introduction).
• Y. Yamada et al., The AI Scientist-v2, 2025: paper.
• A. Ghareeb et al., Robin: A multi-agent system for automating scientific discovery, 2025. paper
• D. A. Boiko et al., Autonomous chemical research with large language models, Nature 2023. paper
• Repositories in the table above; licences read from the GitHub API or LICENSE file on 2026-10-21.