Benchmark 11 of 18, 12 tasks, asked for 53 s to 7 h 40 min
CORE-Bench
Reproducing the computational results of a published research paper, given its code capsule, across fields from archaeology to linguistics.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6.1 SolCodex | 75%18 of 24 runs | 1.04× | 100.012 graded | |
| 2 | GPT 6 AstraCodex | 63%15 of 24 runs | 1.04× | 33.312 graded | |
| 3 | GPT 5.6 SolCodex | 29%7 of 24 runs | 2.02× | 8.312 graded | |
| 4 | Gemini 4 ArgonAntigravity CLI | 8%2 of 24 runs | 2.87× | 75.012 graded | |
| 5 | Claude Fable 5.1Claude Code | 4%1 of 24 runs | 3.33× | 0.012 graded | |
| 6 | Claude Opus 5.5Claude Code | 0%0 of 24 runs | 3.52× | 100.012 graded | |
| 7 | MiMo V2.6 Pro UltraSpeedMiMo Code | 0%0 of 24 runs | 5.08× | 100.012 graded | |
| 8 | MiMo V2.6 ProMiMo Code | 0%0 of 24 runs | 5.65× | 91.712 graded | |
| 9 | Muse Wonder 1.0Muse Code | 0%0 of 24 runs | 7.46× | 91.712 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every CORE-Bench run
Muse Wonder's typical run lasted 1.35× the time asked at the short ask and 0.08× at the long one. Astra's lasted 1.06× and 1.01×.
Hover a mark to read it, click to pin it here.
Showing 216 runs.
The 12 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| Reproduce a rockshelter chronology and archaeological materials analysis | 3.75 min to 7 h | Short 1.03×, long 1.01× | Short 1.28×, long 0.10× | Short 1.58×, long 0.17× | Short 1.18×, long 0.25× | Short 4.81×, long 0.24× | Short 1.02×, long 1.00× | Short 2.22×, long 0.16× | Short 2.27×, long 0.11× | Short 1.93×, long 0.07× |
| Reproduce article and script-availability results across linguistics | 2.5 min to 6 h | Short 1.05×, long 1.00× | Short 1.08×, long 0.07× | Short 0.75×, long 0.07× | Short 0.64×, long 0.15× | Short 4.32×, long 0.18× | Short 1.05×, long 1.01× | Short 2.03×, long 0.10× | Short 2.71×, long 0.16× | Short 4.07×, long 0.10× |
| Rebuild the figures for selective exposure to partisan online news | 3 min to 7.67 h | Short 1.12×, long 1.01× | Short 3.40×, long 1.01× | Short 4.92×, long 0.29× | Short 2.02×, long 0.26× | Short 1.80×, long 1.09× | Short 1.03×, long 1.01× | Short 3.28×, long 0.25× | Short 3.05×, long 0.14× | Short 1.48×, long 0.23× |
| Execute the fibrous-media notebook and reproduce its pore-size result | 3 min to 5.2 h | Short 1.14×, long 1.01× | Short 1.06×, long 1.00× | Short 0.64×, long 0.13× | Short 0.67×, long 0.10× | Short 2.11×, long 0.22× | Short 1.05×, long 1.01× | Short 0.61×, long 0.05× | Short 0.38×, long 0.04× | Short 0.24×, long 0.04× |
| Reproduce journal and university ranking figures | 2 min to 4 h | Short 1.04×, long 1.01× | Short 2.62×, long 1.01× | Short 1.62×, long 0.24× | Short 3.82×, long 0.25× | Short 8.94×, long 1.03× | Short 1.05×, long 1.00× | Short 4.74×, long 0.21× | Short 2.16×, long 0.21× | Short 1.74×, long 0.12× |
| Run a continuous-time intelligent-reflecting-surface propagation model | 52.5 s to 4 h | Short 1.04×, long 1.01× | Short 3.57×, long 0.28× | Short 4.44×, long 0.49× | Short 8.32×, long 0.49× | Short 1.55×, long 1.17× | Short 1.13×, long 1.02× | Short 1.69×, long 0.13× | Short 0.73×, long 0.07× | Short 1.23×, long 0.02× |
| Reproduce a 20 mm transducer-array characterization | 52.5 s to 5.33 h | Short 1.06×, long 1.02× | Short 1.11×, long 1.04× | Short 1.29×, long 0.10× | Short 0.75×, long 0.23× | Short 32.00×, long 2.00× | Short 1.08×, long 1.01× | Short 0.12×, long 0.01× | Short 0.15×, long 0.01× | Short 0.13×, long 0.01× |
| Run a quantum measurement and tomography toolbox | 1.25 min to 4 h | Short 1.08×, long 1.01× | Short 1.89×, long 1.02× | Short 1.04×, long 0.19× | Short 0.72×, long 0.19× | Short 1.78×, long 1.06× | Short 1.11×, long 1.01× | Short 0.54×, long 0.04× | Short 0.71×, long 0.03× | Short 0.36×, long 0.01× |
| Execute relative-entropy causal-influence notebooks in parallel | 2 min to 5.67 h | Short 1.06×, long 1.01× | Short 6.31×, long 1.02× | Short 0.90×, long 0.19× | Short 12.70×, long 0.26× | Short 18.29×, long 1.22× | Short 1.36×, long 1.01× | Short 0.33×, long 0.37× | Short 0.20×, long 0.89× | Short 32.00×, long 0.02× |
| Run product-review sentiment mining and reproduce its metrics | 1.5 min to 5.67 h | Short 1.09×, long 1.11× | Short 4.63×, long 1.22× | Short 6.59×, long 0.56× | Short 17.74×, long 0.64× | Short 22.45×, long 1.46× | Short 1.03×, long 1.00× | Short 2.31×, long 0.12× | Short 3.61×, long 0.19× | Short 2.58×, long 0.22× |
| ESR1 mutant breast cancers show elevated basal cytokeratins and immune activation | 3.5 min to 6 h | No runs | No runs | No runs | Short 0.33×, long 0.07× | Short 1.14×, long 0.25× | Short 1.03×, long 1.00× | Short 1.56×, long 0.07× | Short 0.88×, long 0.19× | Short 0.64×, long 0.09× |
| Human Activities Recognition on PAMAP2 dataset using Random Forest | 2 min to 4.67 h | No runs | No runs | No runs | Short 2.41×, long 0.67× | Short 1.71×, long 1.03× | Short 1.03×, long 1.01× | Short 0.20×, long 0.02× | Short 0.52×, long 0.03× | Short 0.30×, long 0.22× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
The paper’s three agents also ran 2 tasks here that v1.1 replaced: Run and render CNN short-term temperature forecasting results, Train and evaluate end-to-end neural-network feature selection on MNIST.
About CORE-Bench: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by CORE-Bench's own grader, not a scorer AgentTime invented. The pass rule: Every answer correct, with numbers inside a 95% prediction interval.
- Tasks
- 12
- Asked for
- 53 s to 7 h 40 min
- Runs here
- 216
- 9 agents
- Graded as
- Pass or fail: Every answer correct, with numbers inside a 95% prediction interval.
- Source
- Nadgir et al., 2026
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 3 runs here were stopped by AgentTime, so their time is a lower bound. Details
Data and citation
Upstream: Nadgir et al., 2026