Benchmark 16 of 18, 20 tasks, asked for 5 s to 8 h

Humanity's Last Exam

A very hard, wide-ranging exam of expert-level questions across dozens of academic subjects, built to still be difficult once older benchmarks were saturated.

By agent

Agents on Humanity's Last Exam ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6 AstraCodex 50%24 of 48 runs 1.09× 79.224 graded
2 GPT 5.6 SolCodex 48%23 of 48 runs 1.28× 70.824 graded
3 GPT 6.1 SolCodex 43%17 of 40 runs 1.61× 55.020 graded
4 MiMo V2.6 Pro UltraSpeedMiMo Code 13%5 of 40 runs 5.56× 5.020 graded
5 Claude Fable 5.1Claude Code 6%3 of 48 runs 3.04× 87.524 graded
6 Muse Wonder 1.0Muse Code 5%2 of 40 runs 2.44× 35.020 graded
7 Claude Opus 5.5Claude Code 5%2 of 40 runs 2.61× 63.219 graded
8 Gemini 4 ArgonAntigravity CLI 5%2 of 40 runs 3.19× 65.020 graded
9 MiMo V2.6 ProMiMo Code 5%2 of 40 runs 3.91× 5.020 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every Humanity's Last Exam run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

MiMo UltraSpeed's typical run lasted 0.82× the time asked at the short ask and 0.07× at the long one. Astra's lasted 1.13× and 1.01×.

Hover a mark to read it, click to pin it here.

Showing 384 runs.

The 20 tasks

The 20 tasks on Humanity's Last Exam, with where each agent's runs ended
TaskAsked forOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
Item 257.5 s to 1 hShort 3.58×, long 1.47×Short 8.61×, long 1.29×Short 2.06×, long 1.04×Short 1.04×, long 0.17×Short 0.04×, long 0.01×Short 1.66×, long 0.18×
Item 2630 s to 1.67 hShort 3.14×, long 0.21×Short 9.90×, long 1.07×Short 2.22×, long 1.01×Short 8.67×, long 0.83×Short 1.04×, long 0.14×Short 3.47×, long 0.40×
Item 2718 s to 48 minShort 21.67×, long 1.08×Short 128.54×, long 3.07×Short 8.33×, long 1.02×Short 22.98×, long 1.74×Short 2.44×, long 0.17×Short 6.06×, long 0.90×
Item 2837.5 s to 1.33 hShort 4.45×, long 1.40×Short 5.68×, long 1.08×Short 1.18×, long 1.02×Short 8.53×, long 0.47×Short 1.02×, long 0.08×Short 2.59×, long 0.37×
Item 297.5 s to 20 minShort 4.64×, long 1.34×Short 6.51×, long 1.22×Short 3.05×, long 1.05×Short 4.00×, long 0.12×Short 0.20×, long 0.04×Short 3.19×, long 0.29×
Item 3052.5 s to 2.33 hShort 1.11×, long 0.97×Short 1.56×, long 1.04×Short 3.95×, long 1.01×Short 7.77×, long 0.57×Short 0.63×, long 0.09×Short 1.17×, long 0.17×
Item 3152.5 s to 3 hShort 5.61×, long 0.98×Short 1.33×, long 3.34×Short 4.98×, long 1.01×Short 6.23×, long 0.43×Short 0.77×, long 0.05×Short 0.80×, long 0.40×
Item 3237.5 s to 8 hShort 3.76×, long 1.25×Short 3.82×, long 1.09×Short 1.57×, long 1.02×Short 1.22×, long 0.09×Short 0.21×, long 0.03×Short 3.17×, long 0.13×
Item 3352.5 s to 8 hShort 5.19×, long 1.21×Short 4.18×, long 0.97×Short 2.26×, long 1.01×Short 7.71×, long 0.83×Short 0.51×, long 0.02×Short 3.57×, long 2.36×
Item 3437.5 s to 3 hShort 3.30×, long 1.09×Short 7.33×, long 1.08×Short 1.67×, long 1.03×Short 3.24×, long 0.45×Short 0.88×, long 0.03×Short 6.53×, long 0.28×
Item 357.5 s to 20 minShort 3.49×, long 1.18×Short 4.84×, long 1.19×Short 3.05×, long 1.07×Short 20.01×, long 0.22×Short 0.17×, long 0.02×Short 2.25×, long 0.51×
Item 3622.5 s to 48 minShort 1.91×, long 1.06×Short 4.79×, long 1.10×Short 1.55×, long 1.02×Short 9.54×, long 1.04×Short 0.75×, long 0.23×Short 2.27×, long 0.58×
Item 376 s to 20 minShort 4.52×, long 1.16×Short 75.15×, long 1.16×Short 3.55×, long 1.05×Short 12.55×, long 2.82×Short 0.90×, long 0.06×Short 2.85×, long 0.40×
Item 386 s to 4 minShort 20.28×, long 1.99×Short 17.02×, long 1.55×Short 2.45×, long 1.12×Short 7.16×, long 0.51×Short 0.34×, long 0.02×Short 3.05×, long 0.91×
Item 396 s to 12 minShort 2.39×, long 1.49×Short 4.93×, long 1.62×Short 1.65×, long 1.03×Short 6.67×, long 3.94×Short 0.98×, long 1.18×Short 2.36×, long 0.57×
Item 407.5 s to 24 minShort 9.89×, long 1.91×Short 20.19×, long 1.18×Short 2.78×, long 1.04×Short 3.45×, long 0.88×Short 0.55×, long 0.03×Short 1.00×, long 0.77×
Item 414.5 s to 8 minShort 5.82×, long 1.53×Short 7.07×, long 1.11×Short 2.61×, long 1.09×Short 7.45×, long 1.93×Short 1.79×, long 0.18×Short 6.07×, long 1.12×
Item 4210.5 s to 16 minShort 2.35×, long 1.26×Short 3.44×, long 1.11×Short 1.43×, long 1.02×Short 5.10×, long 1.96×Short 2.66×, long 0.16×Short 1.65×, long 0.53×
Item 4312 s to 12 minShort 10.17×, long 2.15×Short 7.50×, long 1.07×Short 3.02×, long 1.04×Short 35.15×, long 0.57×Short 0.98×, long 0.09×Short 1.89×, long 0.41×
Item 447.5 s to 28 minShort 7.64×, long 1.87×Short 16.11×, long 1.25×Short 2.81×, long 1.05×Short 2.65×, long 1.65×Short 1.02×, long 0.12×Short 0.98×, long 0.28×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

The paper’s three agents also ran 24 tasks here that v1.1 replaced: Item 1, Item 2, Item 3, Item 4, Item 5, Item 6, Item 7, Item 8, Item 9, Item 10, Item 11, Item 12, Item 13, Item 14, Item 15, Item 16, Item 17, Item 18, Item 19, Item 20, Item 21, Item 22, Item 23, Item 24.

About Humanity's Last Exam: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by Humanity's Last Exam's own grader, not a scorer AgentTime invented. The pass rule: The benchmark's own judge marks the answer correct.

Tasks
20
Asked for
5 s to 8 h
Runs here
384
9 agents
Graded as
Pass or fail: The benchmark's own judge marks the answer correct.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 2 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Phan et al., 2025