Benchmark 11 of 18, 12 tasks, asked for 53 s to 7 h 40 min

CORE-Bench

Reproducing the computational results of a published research paper, given its code capsule, across fields from archaeology to linguistics.

By agent

Agents on CORE-Bench ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6.1 SolCodex 75%18 of 24 runs 1.04× 100.012 graded
2 GPT 6 AstraCodex 63%15 of 24 runs 1.04× 33.312 graded
3 GPT 5.6 SolCodex 29%7 of 24 runs 2.02× 8.312 graded
4 Gemini 4 ArgonAntigravity CLI 8%2 of 24 runs 2.87× 75.012 graded
5 Claude Fable 5.1Claude Code 4%1 of 24 runs 3.33× 0.012 graded
6 Claude Opus 5.5Claude Code 0%0 of 24 runs 3.52× 100.012 graded
7 MiMo V2.6 Pro UltraSpeedMiMo Code 0%0 of 24 runs 5.08× 100.012 graded
8 MiMo V2.6 ProMiMo Code 0%0 of 24 runs 5.65× 91.712 graded
9 Muse Wonder 1.0Muse Code 0%0 of 24 runs 7.46× 91.712 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every CORE-Bench run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Muse Wonder's typical run lasted 1.35× the time asked at the short ask and 0.08× at the long one. Astra's lasted 1.06× and 1.01×.

Hover a mark to read it, click to pin it here.

Showing 216 runs.

The 12 tasks

The 12 tasks on CORE-Bench, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
Reproduce a rockshelter chronology and archaeological materials analysis3.75 min to 7 hShort 1.03×, long 1.01×Short 1.28×, long 0.10×Short 1.58×, long 0.17×Short 1.18×, long 0.25×Short 4.81×, long 0.24×Short 1.02×, long 1.00×Short 2.22×, long 0.16×Short 2.27×, long 0.11×Short 1.93×, long 0.07×
Reproduce article and script-availability results across linguistics2.5 min to 6 hShort 1.05×, long 1.00×Short 1.08×, long 0.07×Short 0.75×, long 0.07×Short 0.64×, long 0.15×Short 4.32×, long 0.18×Short 1.05×, long 1.01×Short 2.03×, long 0.10×Short 2.71×, long 0.16×Short 4.07×, long 0.10×
Rebuild the figures for selective exposure to partisan online news3 min to 7.67 hShort 1.12×, long 1.01×Short 3.40×, long 1.01×Short 4.92×, long 0.29×Short 2.02×, long 0.26×Short 1.80×, long 1.09×Short 1.03×, long 1.01×Short 3.28×, long 0.25×Short 3.05×, long 0.14×Short 1.48×, long 0.23×
Execute the fibrous-media notebook and reproduce its pore-size result3 min to 5.2 hShort 1.14×, long 1.01×Short 1.06×, long 1.00×Short 0.64×, long 0.13×Short 0.67×, long 0.10×Short 2.11×, long 0.22×Short 1.05×, long 1.01×Short 0.61×, long 0.05×Short 0.38×, long 0.04×Short 0.24×, long 0.04×
Reproduce journal and university ranking figures2 min to 4 hShort 1.04×, long 1.01×Short 2.62×, long 1.01×Short 1.62×, long 0.24×Short 3.82×, long 0.25×Short 8.94×, long 1.03×Short 1.05×, long 1.00×Short 4.74×, long 0.21×Short 2.16×, long 0.21×Short 1.74×, long 0.12×
Run a continuous-time intelligent-reflecting-surface propagation model52.5 s to 4 hShort 1.04×, long 1.01×Short 3.57×, long 0.28×Short 4.44×, long 0.49×Short 8.32×, long 0.49×Short 1.55×, long 1.17×Short 1.13×, long 1.02×Short 1.69×, long 0.13×Short 0.73×, long 0.07×Short 1.23×, long 0.02×
Reproduce a 20 mm transducer-array characterization52.5 s to 5.33 hShort 1.06×, long 1.02×Short 1.11×, long 1.04×Short 1.29×, long 0.10×Short 0.75×, long 0.23×Short 32.00×, long 2.00×Short 1.08×, long 1.01×Short 0.12×, long 0.01×Short 0.15×, long 0.01×Short 0.13×, long 0.01×
Run a quantum measurement and tomography toolbox1.25 min to 4 hShort 1.08×, long 1.01×Short 1.89×, long 1.02×Short 1.04×, long 0.19×Short 0.72×, long 0.19×Short 1.78×, long 1.06×Short 1.11×, long 1.01×Short 0.54×, long 0.04×Short 0.71×, long 0.03×Short 0.36×, long 0.01×
Execute relative-entropy causal-influence notebooks in parallel2 min to 5.67 hShort 1.06×, long 1.01×Short 6.31×, long 1.02×Short 0.90×, long 0.19×Short 12.70×, long 0.26×Short 18.29×, long 1.22×Short 1.36×, long 1.01×Short 0.33×, long 0.37×Short 0.20×, long 0.89×Short 32.00×, long 0.02×
Run product-review sentiment mining and reproduce its metrics1.5 min to 5.67 hShort 1.09×, long 1.11×Short 4.63×, long 1.22×Short 6.59×, long 0.56×Short 17.74×, long 0.64×Short 22.45×, long 1.46×Short 1.03×, long 1.00×Short 2.31×, long 0.12×Short 3.61×, long 0.19×Short 2.58×, long 0.22×
ESR1 mutant breast cancers show elevated basal cytokeratins and immune activation3.5 min to 6 hNo runsNo runsNo runsShort 0.33×, long 0.07×Short 1.14×, long 0.25×Short 1.03×, long 1.00×Short 1.56×, long 0.07×Short 0.88×, long 0.19×Short 0.64×, long 0.09×
Human Activities Recognition on PAMAP2 dataset using Random Forest2 min to 4.67 hNo runsNo runsNo runsShort 2.41×, long 0.67×Short 1.71×, long 1.03×Short 1.03×, long 1.01×Short 0.20×, long 0.02×Short 0.52×, long 0.03×Short 0.30×, long 0.22×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

The paper’s three agents also ran 2 tasks here that v1.1 replaced: Run and render CNN short-term temperature forecasting results, Train and evaluate end-to-end neural-network feature selection on MNIST.

About CORE-Bench: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by CORE-Bench's own grader, not a scorer AgentTime invented. The pass rule: Every answer correct, with numbers inside a 95% prediction interval.

Tasks
12
Asked for
53 s to 7 h 40 min
Runs here
216
9 agents
Graded as
Pass or fail: Every answer correct, with numbers inside a 95% prediction interval.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 3 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Nadgir et al., 2026