Benchmark 7 of 18, 12 tasks, asked for 30 s to 16 h
Agents' Last Exam
Long, economically realistic professional workflows drawn from finance, legal and other industries, like auditing a merger or parsing a stack of SEC filings.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6.1 SolCodex | 71%17 of 24 runs | 1.06× | 64.512 graded | |
| 2 | GPT 6 AstraCodex | 67%16 of 24 runs | 1.14× | 56.412 graded | |
| 3 | GPT 5.6 SolCodex | 29%7 of 24 runs | 2.57× | 64.712 graded | |
| 4 | Gemini 4 ArgonAntigravity CLI | 8%2 of 24 runs | 2.16× | 72.212 graded | |
| 5 | Claude Opus 5.5Claude Code | 0%0 of 24 runs | 2.03× | 67.112 graded | |
| 6 | Claude Fable 5.1Claude Code | 0%0 of 24 runs | 3.71× | 57.312 graded | |
| 7 | MiMo V2.6 ProMiMo Code | 0%0 of 24 runs | 3.97× | 57.912 graded | |
| 8 | MiMo V2.6 Pro UltraSpeedMiMo Code | 0%0 of 24 runs | 10.82× | 52.312 graded | |
| 9 | Muse Wonder 1.0Muse Code | 0%0 of 24 runs | 19.83× | 52.511 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every Agents' Last Exam run
Muse Wonder's typical run lasted 0.16× the time asked at the short ask and 0.01× at the long one. Sol 6.1's lasted 1.07× and 1.01×.
Hover a mark to read it, click to pin it here.
Showing 216 runs.
The 12 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| Redesign and validate a Flowable supply-disruption workflow | 3.75 min to 14 h | Short 1.10×, long 1.01× | Short 7.44×, long 1.01× | Short 4.25×, long 0.53× | Short 5.23×, long 0.74× | Short 4.47×, long 0.92× | Short 1.09×, long 1.01× | Short 0.65×, long 0.05× | Short 0.22×, long 0.01× | Short 0.18×, long 0.04× |
| Parse a fixed 100-filing SEC 10-K corpus | 6.25 min to 16 h | Short 1.29×, long 1.01× | Short 2.25×, long 1.01× | Short 3.95×, long 0.44× | Short 0.91×, long 0.26× | Short 1.54×, long 0.57× | Short 1.04×, long 1.00× | Short 0.91×, long 0.06× | Short 0.35×, long 0.06× | Short 0.67×, long 0.08× |
| Audit four Chinese M&A filings for inconsistencies | 2.75 min to 7 h | Short 1.01×, long 1.00× | Short 1.04×, long 0.06× | Short 1.40×, long 0.14× | Short 0.61×, long 0.13× | Short 2.43×, long 1.03× | Short 1.03×, long 1.01× | Short 1.20×, long 0.09× | Short 0.13×, long 0.01× | Short 0.03×, long 0.00× |
| Repair a Moodle backup and rebuild registrar exports | 3.75 min to 6.33 h | Short 1.64×, long 1.01× | Short 1.92×, long 0.10× | Short 31.04×, long 0.22× | Short 0.85×, long 0.18× | Short 1.23×, long 0.17× | Short 1.05×, long 1.01× | Short 0.50×, long 0.07× | Short 0.11×, long 0.01× | Short 0.08×, long 0.01× |
| Remediate MARC records and prepare a FOLIO overlay | 5 min to 6 h | Short 1.17×, long 1.00× | Short 4.93×, long 1.00× | Short 4.22×, long 0.41× | Short 0.77×, long 0.54× | Short 2.54×, long 0.50× | Short 1.02×, long 1.01× | Short 1.09×, long 0.13× | Short 0.25×, long 0.03× | Short 0.06×, long 0.01× |
| Build a clean retail SQLite warehouse from messy inputs | 3.5 min to 4.13 h | Short 1.03×, long 1.00× | Short 1.23×, long 1.08× | Short 2.63×, long 0.28× | Short 1.10×, long 0.31× | Short 1.90×, long 0.27× | Short 1.08×, long 1.01× | Short 0.45×, long 0.03× | Short 0.20×, long 0.02× | Short 0.14×, long 0.01× |
| Repair and calibrate an urban SUMO traffic simulation | 5 min to 11 h | Short 1.84×, long 1.00× | Short 5.62×, long 1.01× | Short 23.05×, long 0.42× | Short 0.24×, long 0.42× | Short 1.29×, long 0.58× | Short 1.02×, long 1.01× | Short 0.91×, long 0.08× | Short 0.31×, long 0.03× | Short 0.25×, long 0.02× |
| Build a reusable HST ACS/WFC visit-reduction pipeline | 5 min to 11 h | Short 1.66×, long 1.01× | Short 3.52×, long 1.01× | Short 9.54×, long 0.48× | Short 0.71×, long 0.80× | Short 3.24×, long 0.32× | Short 1.04×, long 1.01× | Short 0.63×, long 0.08× | Short 0.27×, long 0.03× | Short 0.07×, long 0.01× |
| Find near-best-known routes for three CVRP instances | 2 min to 6.67 h | Short 1.18×, long 1.03× | Short 2.89×, long 1.24× | Short 1.34×, long 0.77× | Short 0.70×, long 0.88× | Short 3.23×, long 0.85× | Short 1.13×, long 1.02× | Short 1.79×, long 0.83× | Short 1.92×, long 0.40× | Short 0.73×, long 0.37× |
| Apple FY2024 Balance Sheet Reconstruction | 30 s to 48 min | No runs | No runs | No runs | Short 1.36×, long 0.38× | Short 4.12×, long 0.39× | Short 1.58×, long 1.04× | Short 0.52×, long 0.10× | Short 0.34×, long 0.06× | Short 0.36×, long 0.03× |
| Chisel-to-Verilog Source Location Alignment | 1.5 min to 3.67 h | No runs | No runs | No runs | Short 1.17×, long 0.42× | Short 1.68×, long 1.03× | Short 1.14×, long 1.02× | Short 0.62×, long 0.12× | Short 0.35×, long 0.05× | Short 0.12×, long 0.01× |
| Molecular Structure Plausibility | 2 min to 2.67 h | No runs | No runs | No runs | Short 0.50×, long 0.47× | Short 7.46×, long 1.06× | Short 1.09×, long 1.01× | Short 1.69×, long 0.13× | Short 0.42×, long 0.03× | Short 0.21×, long 0.02× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
The paper’s three agents also ran 3 tasks here that v1.1 replaced: Complete an end-to-end Odoo supply-chain workflow, Route a KiCad PCB from the staged schematic, Compress a 3D Gaussian Splatting scene without losing quality.
About Agents' Last Exam: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by Agents' Last Exam's own grader, not a scorer AgentTime invented. The pass rule: Full pass: a checker score of 1.
- Tasks
- 12
- Asked for
- 30 s to 16 h
- Runs here
- 216
- 9 agents
- Graded as
- Pass or fail: Full pass: a checker score of 1.
- Source
- Sun et al., 2026
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 6 runs here were stopped by AgentTime, so their time is a lower bound. Details
Data and citation
Upstream: Sun et al., 2026