Benchmark 12 of 18, 4 tasks, asked for 5 min to 97 h 36 min
PaperBench
Reproducing a machine learning research paper from scratch, using only the paper itself and its addendum, no original code.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark |
|---|---|---|---|---|
| 1 | GPT 6 AstraCodex | 100%6 of 6 runs | 1.01× | |
| 2 | GPT 6.1 SolCodex | 75%6 of 8 runs | 1.13× | |
| 3 | GPT 5.6 SolCodex | 33%2 of 6 runs | 1.36× | |
| 4 | MiMo V2.6 ProMiMo Code | 14%1 of 7 runs | 2.14× | |
| 5 | Claude Opus 5.5Claude Code | 0%0 of 6 runs | 2.12× | |
| 6 | Gemini 4 ArgonAntigravity CLI | 0%0 of 8 runs | 2.22× | |
| 7 | Claude Fable 5.1Claude Code | 0%0 of 6 runs | 2.63× | |
| 8 | MiMo V2.6 Pro UltraSpeedMiMo Code | 0%0 of 8 runs | 3.67× |
PaperBench is not in the benchmark score. How it is measured
Every PaperBench run
MiMo UltraSpeed's typical run lasted 1.98× the time asked at the short ask and 0.48× at the long one. Astra's lasted 1.02× and 1.00×.
Hover a mark to read it, click to pin it here.
Showing 55 runs.
The 4 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed |
|---|---|---|---|---|---|---|---|---|---|
| Bridging Data Gaps | 12.5 min to 60 h | Short 1.01×, long 1.00× | Short 0.49×, long 1.00× | Short 0.42×, long 0.23× | No runs | Short 1.56×, long 0.23× | Short 1.02×, long 0.87× | Short 1.52×, long 0.86× | Short 4.18×, long 0.01× |
| FRE | 10 min to 97.6 h | Short 1.02×, long 1.00× | Short 0.64×, long 1.00× | Short 0.50×, long 0.28× | Short 0.92×, long 0.39× | Short 1.79×, long 0.73× | Short 1.02×, long 2.00× | Short 0.35×, long 0.26× | Short 0.27×, long 0.68× |
| BAM | 5 min to 43.9 h | Short 1.03×, long 1.01× | Short 1.73×, long 1.14× | Short 1.25×, long 0.28× | Short 0.54×, long 0.26× | Short 2.04×, long 0.39× | Short 1.02×, long 1.05× | Short 0.96×, long 0.11× | Short 1.97×, long 0.60× |
| Challenges in Training PINNs: A Loss Landscape Perspective | 15.8 min to 55.4 h | No runs | No runs | No runs | Short 0.75×, long 0.29× | Short 3.79×, long 0.56× | Short 1.01×, long 1.00× | Short 0.88×, long no run | Short 1.98×, long 0.37× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
About PaperBench: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by PaperBench's own grader, not a scorer AgentTime invented. It reports a score: Replication score: a weighted rubric of pass/fail leaf checks, 0 to 1, after a full reproduction attempt.
- Tasks
- 4
- Asked for
- 5 min to 97 h 36 min
- Runs here
- 55
- 8 agents
- Graded as
- Replication score: a weighted rubric of pass/fail leaf checks, 0 to 1, after a full reproduction attempt.
- Source
- Starace et al., 2025
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 1 run here was stopped by AgentTime, so its time is a lower bound. Details
Data and citation
Upstream: Starace et al., 2025