Benchmark 12 of 18, 4 tasks, asked for 5 min to 97 h 36 min

PaperBench

Reproducing a machine learning research paper from scratch, using only the paper itself and its addendum, no original code.

By agent

Agents on PaperBench ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmark
1 GPT 6 AstraCodex 100%6 of 6 runs 1.01×
2 GPT 6.1 SolCodex 75%6 of 8 runs 1.13×
3 GPT 5.6 SolCodex 33%2 of 6 runs 1.36×
4 MiMo V2.6 ProMiMo Code 14%1 of 7 runs 2.14×
5 Claude Opus 5.5Claude Code 0%0 of 6 runs 2.12×
6 Gemini 4 ArgonAntigravity CLI 0%0 of 8 runs 2.22×
7 Claude Fable 5.1Claude Code 0%0 of 6 runs 2.63×
8 MiMo V2.6 Pro UltraSpeedMiMo Code 0%0 of 8 runs 3.67×

PaperBench is not in the benchmark score. How it is measured

Every PaperBench run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

MiMo UltraSpeed's typical run lasted 1.98× the time asked at the short ask and 0.48× at the long one. Astra's lasted 1.02× and 1.00×.

Hover a mark to read it, click to pin it here.

Showing 55 runs.

The 4 tasks

The 4 tasks on PaperBench, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeed
Bridging Data Gaps12.5 min to 60 hShort 1.01×, long 1.00×Short 0.49×, long 1.00×Short 0.42×, long 0.23×No runsShort 1.56×, long 0.23×Short 1.02×, long 0.87×Short 1.52×, long 0.86×Short 4.18×, long 0.01×
FRE10 min to 97.6 hShort 1.02×, long 1.00×Short 0.64×, long 1.00×Short 0.50×, long 0.28×Short 0.92×, long 0.39×Short 1.79×, long 0.73×Short 1.02×, long 2.00×Short 0.35×, long 0.26×Short 0.27×, long 0.68×
BAM5 min to 43.9 hShort 1.03×, long 1.01×Short 1.73×, long 1.14×Short 1.25×, long 0.28×Short 0.54×, long 0.26×Short 2.04×, long 0.39×Short 1.02×, long 1.05×Short 0.96×, long 0.11×Short 1.97×, long 0.60×
Challenges in Training PINNs: A Loss Landscape Perspective15.8 min to 55.4 hNo runsNo runsNo runsShort 0.75×, long 0.29×Short 3.79×, long 0.56×Short 1.01×, long 1.00×Short 0.88×, long no runShort 1.98×, long 0.37×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

About PaperBench: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by PaperBench's own grader, not a scorer AgentTime invented. It reports a score: Replication score: a weighted rubric of pass/fail leaf checks, 0 to 1, after a full reproduction attempt.

Tasks
4
Asked for
5 min to 97 h 36 min
Runs here
55
8 agents
Graded as
Replication score: a weighted rubric of pass/fail leaf checks, 0 to 1, after a full reproduction attempt.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 1 run here was stopped by AgentTime, so its time is a lower bound. Details

Data and citation

Upstream: Starace et al., 2025