Benchmark 2 of 18, 28 tasks, asked for 1 min 15 s to 20 min
GPQA Diamond
Graduate-level, multiple-choice science questions in physics and chemistry, written so that a quick web search will not give away the answer.
By agent
| Rank | Agent | Timing erroron this benchmark | Where runs ended, from a tenth of the time asked to ten times it | Within 5%0.95× to 1.05× of the request | Scoreout of 100 | Pass rateofficial rule |
|---|---|---|---|---|---|---|
| 1 | GPT 6 AstraCodex | 1.06× | 65%55 of 84 runs | 96.4 | 96.4%81 of 84 passed | |
| 2 | GPT 5.6 SolCodex | 1.34× | 46%39 of 84 runs | 94.0 | 94%79 of 84 passed | |
| 3 | Claude Fable 5.1Claude Code | 3.76× | 5%4 of 84 runs | 85.7 | 85.7%72 of 84 passed | |
| Muse Spark 1.3Muse Code. Partial, not ranked | 5.66× | 1%1 of 84 runs | 95.974 of 84 graded, not the paper's | 95.9%71 of 74 passed |
Score is the paper's mean over each agent's graded runs, out of 100. Pass rate follows the official rule.
How to read this board
Where runs ended counts every run from a tenth of the time asked to ten times it. Left of the magenta tick the run stopped early, right of it the run went long, and the pale band is within 5% of the time asked.
Score is the paper's mean over each agent's graded runs, out of 100. GPQA Diamond counts toward the benchmark score.
Pass rate is the share of graded runs that met the official rule: The answer letter matches the key.
326 of 336 runs graded. 325 graded during the run and 1 left nothing to grade (counted as 0). Not graded: 10 no grade recorded.
Muse Spark 1.3 is a partial run: shown, never ranked against the paper's three agents, and its grades are the campaign's own, not the paper's. How runs are graded
Every GPQA Diamond run
Fable's typical run lasted 1 min 35 s when asked for 1 min 15 s and 1 min 8 s when asked for 20 min. Astra's lasted 1 min 23 s and 20 min 11 s.
Hover a mark to read it, click to pin it here.
Showing 252 runs.
The 28 tasks
| Task | Asked for | Astra | Sol | Fable |
|---|---|---|---|---|
| Question 1 (Chemistry) | 1.25, 5, 20 min | 1.6, 5.2, 20.1 min | 9.9, 9.9, 20.4 min | 2.7, 7, 22.2 min |
| Question 2 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.1, 20.2 min | 1.4, 5.3, 20.3 min | 1.6 min, 35 s, 40 s |
| Question 3 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.7, 5.2, 20.4 min | 1.6, 1.6, 1.4 min |
| Question 4 (Chemistry) | 1.25, 5, 20 min | 1.6, 5.1, 20.2 min | 4.2, 5.3, 20.4 min | 2.3, 8.5, 5.4 min |
| Question 5 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.4, 5.3, 20.3 min | 1.6 min, 1 min, 46 s |
| Question 6 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 20.3 min | 4.1, 5.2, 20.1 min | 1.7, 9.8, 6.6 min |
| Question 7 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.1, 20.2 min | 1.5, 5.2, 20.4 min | 1.5, 2.5, 1.5 min |
| Question 8 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 2.3, 5.2, 20.3 min | 2, 5.2, 1.9 min |
| Question 9 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 20.3 min | 2, 5.3, 20.2 min | 3.8, 1.4, 2.2 min |
| Question 10 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.1, 20.2 min | 1.4, 5.1, 20.3 min | 1.5 min, 50 s, 20.3 min |
| Question 11 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 20.3 min | 1.6, 5.4, 20.2 min | 1.7, 5.6, 1.6 min |
| Question 12 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 20.1 min | 2, 5.2, 20.2 min | 1.8, 2.4, 1.7 min |
| Question 13 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.2, 15.5 min | 3.8, 5.4, 20.2 min | 2.1, 6.6, 17.5 min |
| Question 14 (Chemistry) | 1.25, 5, 20 min | 1.4, 5.1, 20.2 min | 1.5, 5.3, 20.5 min | 1.6, 2.2, 1.1 min |
| Question 15 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.3 min | 1.5, 5.4, 20.5 min | 1.5 min, 51 s, 28 s |
| Question 16 (Physics) | 1.25, 5, 20 min | 1.4, 5.1, 20.2 min | 1.5, 5.2, 20.3 min | 1.6, 2.1, 1.2 min |
| Question 17 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.5 min, 5.2 min, 37 s | 1.7 min, 53 s, 47 s |
| Question 18 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.5 min, 5.3 min, 19 s | 1.6 min, 52 s, 1.6 min |
| Question 19 (Physics) | 1.25, 5, 20 min | 1.4, 5.1, 20.2 min | 1.5, 5.3, 20.4 min | 1.3 min, 43 s, 29 s |
| Question 20 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.4 min | 1.5, 6.5, 20.4 min | 1.5 min, 1.1 min, 47 s |
| Question 21 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.5, 5.3, 20.3 min | 1.5 min, 47 s, 58 s |
| Question 22 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.5, 5.4, 20.3 min | 1.5 min, 47 s, 35 s |
| Question 23 (Physics) | 1.25, 5, 20 min | 1.4, 5.1, 20.2 min | 1.5, 5.2, 20.2 min | 1.5 min, 5.3 min, 49 s |
| Question 24 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.5, 5.2, 20.3 min | 1.5, 5.4, 20.4 min |
| Question 25 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.6 min, 5.2 min, 34 s | 1.5 min, 37 s, 50 s |
| Question 26 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.5, 5.3, 20.3 min | 1.6 min, 1.6 min, 56 s |
| Question 27 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.4, 5.2, 20.4 min | 1.5 min, 33 s, 32 s |
| Question 28 (Physics) | 1.25, 5, 20 min | 1.4, 5.2, 20.2 min | 1.5, 5.2, 20.4 min | 1.5 min, 49 s, 33 s |
No runs match these filters.
Every task is a GPQA Diamond closed-book question. Each mark is one run, at one of the three requests, placed from 0.1× to 10× of the time asked for; the pale band is within 5% of it.
About GPQA Diamond: the prompt, the grader, the source
One GPQA Diamond task, at its longest request, exactly as the agent read the end of its prompt:
Please work on this task for a full 20 minutes. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by GPQA Diamond's own grader, not a scorer AgentTime invented. The pass rule: The answer letter matches the key.
- Tasks
- 28
- Asked for
- 1 min 15 s to 20 min
- Runs here
- 336
- 4 agents
- Graded as
- Pass or fail: The answer letter matches the key.
- Source
- Rein et al., 2024 (COLM)
- Data release
- 26 Sep 2026
- 13:11 UTC
Read before quoting: 2 notes
- GPQA Diamond is meant to be closed-book, but it is one of two benchmarks where the agent is given a callable clock tool, so it can keep track of the requested time.
- For 10 runs on GPQA Diamond, how the run ended has not been labelled yet. They are timed like every other run and counted in every number. Details
Data and citation
Upstream: Rein et al., 2024 (COLM)