Benchmark 17 of 18, 12 tasks, asked for 2 s to 1 h 20 min
GPQA Diamond
Graduate-level, multiple-choice science questions in physics and chemistry, written so that a quick web search will not give away the answer.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6 AstraCodex | 48%27 of 56 runs | 1.07× | 96.428 graded | |
| 2 | GPT 5.6 SolCodex | 45%25 of 56 runs | 1.50× | 96.428 graded | |
| 3 | GPT 6.1 SolCodex | 17%4 of 24 runs | 1.66× | 100.012 graded | |
| 4 | Claude Fable 5.1Claude Code | 5%3 of 56 runs | 4.05× | 85.728 graded | |
| 5 | Claude Opus 5.5Claude Code | 0%0 of 24 runs | 1.93× | 91.712 graded | |
| 6 | Muse Wonder 1.0Muse Code | 0%0 of 24 runs | 2.64× | 83.312 graded | |
| 7 | Gemini 4 ArgonAntigravity CLI | 0%0 of 24 runs | 3.13× | 100.012 graded | |
| 8 | MiMo V2.6 ProMiMo Code | 0%0 of 24 runs | 4.67× | 91.712 graded | |
| 9 | MiMo V2.6 Pro UltraSpeedMiMo Code | 0%0 of 24 runs | 14.56× | 91.712 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every GPQA Diamond run
MiMo UltraSpeed's typical run lasted 0.25× the time asked at the short ask and 0.02× at the long one. Astra's lasted 1.11× and 1.01×.
Hover a mark to read it, click to pin it here.
Showing 312 runs.
The 12 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| Question 1 (Chemistry) | 7.5 s to 32 min | Short 1.31×, long 1.01× | Short 7.94×, long 1.02× | Short 2.13×, long 1.11× | Short 3.63×, long 1.09× | Short 6.87×, long 1.22× | Short 4.45×, long 1.02× | Short 33.10×, long 2.64× | Short 2.27×, long 0.26× | Short 3.25×, long 0.36× |
| Question 4 (Chemistry) | 10.5 s to 1.33 h | Short 1.29×, long 1.01× | Short 3.35×, long 1.02× | Short 1.86×, long 0.27× | Short 5.79×, long 1.86× | Short 7.54×, long 1.45× | Short 1.14×, long 1.02× | Short 13.47×, long 0.42× | Short 0.49×, long 0.07× | Short 4.06×, long 0.65× |
| Question 8 (Chemistry) | 7.5 s to 20 min | Short 1.12×, long 1.01× | Short 1.84×, long 1.02× | Short 1.62×, long 0.09× | Short 3.27×, long 1.18× | Short 7.80×, long 1.23× | Short 1.58×, long 1.03× | Short 4.49×, long 1.18× | Short 0.43×, long 0.20× | Short 2.20×, long 0.15× |
| Question 9 (Chemistry) | 7.5 s to 24 min | Short 1.11×, long 1.02× | Short 1.62×, long 1.01× | Short 3.05×, long 0.11× | Short 1.87×, long 1.06× | Short 9.24×, long 1.13× | Short 3.34×, long 1.03× | Short 3.61×, long 1.29× | Short 1.19×, long 0.08× | Short 2.83×, long 0.39× |
| Question 10 (Chemistry) | 3 s to 20 min | Short 1.11×, long 1.01× | Short 1.15×, long 1.01× | Short 1.18×, long 1.02× | Short 2.82×, long 1.16× | Short 9.03×, long 1.31× | Short 2.29×, long 1.11× | Short 0.87×, long 0.18× | Short 0.10×, long 0.01× | Short 1.55×, long 0.26× |
| Question 12 (Chemistry) | 7.5 s to 20 min | Short 1.09×, long 1.01× | Short 1.57×, long 1.01× | Short 1.46×, long 0.08× | Short 2.48×, long 1.18× | Short 5.99×, long 1.13× | Short 3.94×, long 1.11× | Short 2.17×, long 0.18× | Short 0.37×, long 0.04× | Short 1.71×, long 0.23× |
| Question 15 (Physics) | 4.5 s to 32 min | Short 1.09×, long 1.02× | Short 1.24×, long 1.03× | Short 1.20×, long 0.02× | Short 2.75×, long 1.11× | Short 6.02×, long 1.19× | Short 2.08×, long 1.08× | Short 0.47×, long 0.04× | Short 0.04×, long 0.01× | Short 1.73×, long 0.32× |
| Question 16 (Physics) | 4.5 s to 20 min | Short 1.14×, long 1.01× | Short 1.16×, long 1.02× | Short 1.30×, long 0.06× | Short 3.93×, long 1.18× | Short 8.18×, long 1.16× | Short 1.64×, long 1.06× | Short 0.68×, long 0.15× | Short 0.35×, long 0.06× | Short 1.77×, long 0.27× |
| Question 20 (Physics) | 3 s to 20 min | Short 1.11×, long 1.02× | Short 1.23×, long 1.02× | Short 1.21×, long 0.04× | Short 4.09×, long 1.27× | Short 9.43×, long 1.39× | Short 3.49×, long 1.12× | Short 0.46×, long 0.04× | Short 0.15×, long 0.01× | Short 2.20×, long 0.34× |
| Question 23 (Physics) | 3 s to 20 min | Short 1.09×, long 1.01× | Short 1.23×, long 1.01× | Short 1.23×, long 0.04× | Short 2.80×, long 1.12× | Short 6.47×, long 1.23× | Short 2.82×, long 1.17× | Short 0.44×, long 0.02× | Short 0.10×, long 0.01× | Short 0.31×, long 0.62× |
| Question 25 (Physics) | 3 s to 20 min | Short 1.12×, long 1.01× | Short 1.27×, long 0.03× | Short 1.18×, long 0.04× | Short 2.80×, long 1.13× | Short 9.17×, long 1.25× | Short 2.72×, long 1.20× | Short 0.63×, long 0.06× | Short 0.05×, long 0.00× | Short 1.63×, long 0.28× |
| Question 27 (Physics) | 2.25 s to 20 min | Short 1.10×, long 1.01× | Short 1.13×, long 1.02× | Short 1.23×, long 0.03× | Short 2.88×, long 1.11× | Short 9.11×, long 1.46× | Short 3.22×, long 1.16× | Short 0.51×, long 0.07× | Short 0.06×, long 0.01× | Short 1.64×, long 0.24× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
The paper’s three agents also ran 16 tasks here that v1.1 replaced: Question 2 (Chemistry), Question 3 (Chemistry), Question 5 (Chemistry), Question 6 (Chemistry), Question 7 (Chemistry), Question 11 (Chemistry), Question 13 (Chemistry), Question 14 (Chemistry), Question 17 (Physics), Question 18 (Physics), Question 19 (Physics), Question 21 (Physics), Question 22 (Physics), Question 24 (Physics), Question 26 (Physics), Question 28 (Physics).
About GPQA Diamond: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by GPQA Diamond's own grader, not a scorer AgentTime invented. The pass rule: The answer letter matches the key.
- Tasks
- 12
- Asked for
- 2 s to 1 h 20 min
- Runs here
- 312
- 9 agents
- Graded as
- Pass or fail: The answer letter matches the key.
- Source
- Rein et al., 2024 (COLM)
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 1 run here was stopped by AgentTime, so its time is a lower bound. Details
Data and citation
Upstream: Rein et al., 2024 (COLM)