Benchmark 7 of 18, 12 tasks, asked for 30 s to 16 h

Agents' Last Exam

Long, economically realistic professional workflows drawn from finance, legal and other industries, like auditing a merger or parsing a stack of SEC filings.

By agent

Agents on Agents' Last Exam ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6.1 SolCodex 71%17 of 24 runs 1.06× 64.512 graded
2 GPT 6 AstraCodex 67%16 of 24 runs 1.14× 56.412 graded
3 GPT 5.6 SolCodex 29%7 of 24 runs 2.57× 64.712 graded
4 Gemini 4 ArgonAntigravity CLI 8%2 of 24 runs 2.16× 72.212 graded
5 Claude Opus 5.5Claude Code 0%0 of 24 runs 2.03× 67.112 graded
6 Claude Fable 5.1Claude Code 0%0 of 24 runs 3.71× 57.312 graded
7 MiMo V2.6 ProMiMo Code 0%0 of 24 runs 3.97× 57.912 graded
8 MiMo V2.6 Pro UltraSpeedMiMo Code 0%0 of 24 runs 10.82× 52.312 graded
9 Muse Wonder 1.0Muse Code 0%0 of 24 runs 19.83× 52.511 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every Agents' Last Exam run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Muse Wonder's typical run lasted 0.16× the time asked at the short ask and 0.01× at the long one. Sol 6.1's lasted 1.07× and 1.01×.

Hover a mark to read it, click to pin it here.

Showing 216 runs.

The 12 tasks

The 12 tasks on Agents' Last Exam, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
Redesign and validate a Flowable supply-disruption workflow3.75 min to 14 hShort 1.10×, long 1.01×Short 7.44×, long 1.01×Short 4.25×, long 0.53×Short 5.23×, long 0.74×Short 4.47×, long 0.92×Short 1.09×, long 1.01×Short 0.65×, long 0.05×Short 0.22×, long 0.01×Short 0.18×, long 0.04×
Parse a fixed 100-filing SEC 10-K corpus6.25 min to 16 hShort 1.29×, long 1.01×Short 2.25×, long 1.01×Short 3.95×, long 0.44×Short 0.91×, long 0.26×Short 1.54×, long 0.57×Short 1.04×, long 1.00×Short 0.91×, long 0.06×Short 0.35×, long 0.06×Short 0.67×, long 0.08×
Repair a Moodle backup and rebuild registrar exports3.75 min to 6.33 hShort 1.64×, long 1.01×Short 1.92×, long 0.10×Short 31.04×, long 0.22×Short 0.85×, long 0.18×Short 1.23×, long 0.17×Short 1.05×, long 1.01×Short 0.50×, long 0.07×Short 0.11×, long 0.01×Short 0.08×, long 0.01×
Remediate MARC records and prepare a FOLIO overlay5 min to 6 hShort 1.17×, long 1.00×Short 4.93×, long 1.00×Short 4.22×, long 0.41×Short 0.77×, long 0.54×Short 2.54×, long 0.50×Short 1.02×, long 1.01×Short 1.09×, long 0.13×Short 0.25×, long 0.03×Short 0.06×, long 0.01×
Build a clean retail SQLite warehouse from messy inputs3.5 min to 4.13 hShort 1.03×, long 1.00×Short 1.23×, long 1.08×Short 2.63×, long 0.28×Short 1.10×, long 0.31×Short 1.90×, long 0.27×Short 1.08×, long 1.01×Short 0.45×, long 0.03×Short 0.20×, long 0.02×Short 0.14×, long 0.01×
Repair and calibrate an urban SUMO traffic simulation5 min to 11 hShort 1.84×, long 1.00×Short 5.62×, long 1.01×Short 23.05×, long 0.42×Short 0.24×, long 0.42×Short 1.29×, long 0.58×Short 1.02×, long 1.01×Short 0.91×, long 0.08×Short 0.31×, long 0.03×Short 0.25×, long 0.02×
Build a reusable HST ACS/WFC visit-reduction pipeline5 min to 11 hShort 1.66×, long 1.01×Short 3.52×, long 1.01×Short 9.54×, long 0.48×Short 0.71×, long 0.80×Short 3.24×, long 0.32×Short 1.04×, long 1.01×Short 0.63×, long 0.08×Short 0.27×, long 0.03×Short 0.07×, long 0.01×
Find near-best-known routes for three CVRP instances2 min to 6.67 hShort 1.18×, long 1.03×Short 2.89×, long 1.24×Short 1.34×, long 0.77×Short 0.70×, long 0.88×Short 3.23×, long 0.85×Short 1.13×, long 1.02×Short 1.79×, long 0.83×Short 1.92×, long 0.40×Short 0.73×, long 0.37×
Apple FY2024 Balance Sheet Reconstruction30 s to 48 minNo runsNo runsNo runsShort 1.36×, long 0.38×Short 4.12×, long 0.39×Short 1.58×, long 1.04×Short 0.52×, long 0.10×Short 0.34×, long 0.06×Short 0.36×, long 0.03×
Chisel-to-Verilog Source Location Alignment1.5 min to 3.67 hNo runsNo runsNo runsShort 1.17×, long 0.42×Short 1.68×, long 1.03×Short 1.14×, long 1.02×Short 0.62×, long 0.12×Short 0.35×, long 0.05×Short 0.12×, long 0.01×
Molecular Structure Plausibility2 min to 2.67 hNo runsNo runsNo runsShort 0.50×, long 0.47×Short 7.46×, long 1.06×Short 1.09×, long 1.01×Short 1.69×, long 0.13×Short 0.42×, long 0.03×Short 0.21×, long 0.02×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

The paper’s three agents also ran 3 tasks here that v1.1 replaced: Complete an end-to-end Odoo supply-chain workflow, Route a KiCad PCB from the staged schematic, Compress a 3D Gaussian Splatting scene without losing quality.

About Agents' Last Exam: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by Agents' Last Exam's own grader, not a scorer AgentTime invented. The pass rule: Full pass: a checker score of 1.

Tasks
12
Asked for
30 s to 16 h
Runs here
216
9 agents
Graded as
Pass or fail: Full pass: a checker score of 1.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 6 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Sun et al., 2026