Benchmark 14 of 18, 10 tasks, asked for 6 min 15 s to 28 h

Sakana ALE-Bench

Competitive-programming optimization problems (drawn from AtCoder Heuristic Contests) with no known perfect answer, where the agent keeps improving its own solution.

By agent

Agents on Sakana ALE-Bench ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreown units
1 GPT 6 AstraCodex 95%19 of 20 runs 1.02× Own units
2 GPT 6.1 SolCodex 95%19 of 20 runs 1.02× Own units
3 GPT 5.6 SolCodex 65%13 of 20 runs 1.16× Own units
4 Gemini 4 ArgonAntigravity CLI 25%5 of 20 runs 1.84× Own units
5 Claude Fable 5.1Claude Code 10%2 of 20 runs 1.95× Own units
6 MiMo V2.6 Pro UltraSpeedMiMo Code 5%1 of 20 runs 4.53× Own units
7 Claude Opus 5.5Claude Code 0%0 of 20 runs 1.86× Own units
8 MiMo V2.6 ProMiMo Code 0%0 of 20 runs 3.35× Own units
9 Muse Wonder 1.0Muse Code 0%0 of 20 runs 23.39× Own units

Sakana ALE-Bench keeps its own units, so it has no score out of 100. How it is measured

Every Sakana ALE-Bench run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Muse Wonder's typical run lasted 0.07× the time asked at the short ask and 0.01× at the long one. Astra's lasted 1.02× and 1.00×.

Hover a mark to read it, click to pin it here.

Showing 180 runs.

The 10 tasks

The 10 tasks on Sakana ALE-Bench, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
AtCoder Heuristic Contest 76.25 min to 15 hShort 1.02×, long 1.00×Short 1.29×, long 1.00×Short 1.01×, long 0.42×Short 0.89×, long 0.80×Short 1.67×, long 0.96×Short 1.03×, long 1.01×Short 1.17×, long 0.19×Short 1.04×, long 0.24×Short 0.29×, long 0.01×
AtCoder Heuristic Contest 87.5 min to 28 hShort 1.02×, long 1.00×Short 1.05×, long 1.00×Short 0.50×, long 0.27×Short 0.40×, long 0.30×Short 1.13×, long 0.11×Short 1.02×, long 1.00×Short 0.62×, long 0.07×Short 0.19×, long 0.07×Short 0.62×, long 0.03×
AtCoder Heuristic Contest 96.25 min to 16 hShort 1.02×, long 1.00×Short 1.10×, long 1.01×Short 0.53×, long 0.55×Short 0.37×, long 0.82×Short 0.41×, long 1.00×Short 1.04×, long 1.01×Short 1.16×, long 0.21×Short 1.19×, long 0.05×Short 0.07×, long 0.01×
AtCoder Heuristic Contest 1111.25 min to 25.3 hShort 1.02×, long 1.00×Short 1.03×, long 1.00×Short 0.99×, long 0.30×Short 0.75×, long 0.79×Short 1.08×, long 0.41×Short 1.04×, long 1.00×Short 0.74×, long 0.10×Short 0.38×, long 0.16×Short 0.07×, long 0.01×
Future Contest 2022 Qual10 min to 24 hShort 1.03×, long 1.00×Short 1.03×, long 1.01×Short 0.60×, long 0.28×Short 0.83×, long 0.55×Short 1.40×, long 1.00×Short 1.02×, long 1.00×Short 1.83×, long 0.14×Short 0.58×, long 0.12×Short 0.09×, long 0.04×
AtCoder Heuristic Contest 1610 min to 18 hShort 1.02×, long 1.01×Short 1.81×, long 1.00×Short 0.36×, long 0.40×Short 0.24×, long 0.33×Short 1.11×, long 0.83×Short 1.04×, long 1.00×Short 0.77×, long 0.14×Short 0.79×, long 0.06×Short 0.88×, long 0.06×
AtCoder Heuristic Contest 266.25 min to 20 hShort 1.02×, long 1.01×Short 1.53×, long 1.00×Short 0.78×, long 0.48×Short 0.41×, long 0.37×Short 1.02×, long 1.01×Short 1.03×, long 1.01×Short 0.73×, long 0.09×Short 0.81×, long 0.04×Short 0.07×, long 0.04×
AtCoder Heuristic Contest 3110 min to 20 hShort 1.11×, long 1.00×Short 1.09×, long 1.05×Short 0.67×, long 0.51×Short 0.39×, long 0.84×Short 1.11×, long 0.12×Short 1.03×, long 1.00×Short 0.68×, long 0.12×Short 0.35×, long 0.14×Short 0.03×, long 0.01×
AtCoder Heuristic Contest 398.75 min to 20 hShort 1.04×, long 1.00×Short 1.24×, long 1.02×Short 0.60×, long 0.49×Short 0.75×, long 0.83×Short 1.15×, long 0.02×Short 1.03×, long 1.00×Short 0.76×, long 0.06×Short 0.25×, long 0.19×Short 0.06×, long 0.01×
AtCoder Heuristic Contest 467.5 min to 11 hShort 1.01×, long 1.00×Short 2.89×, long 1.01×Short 0.70×, long 2.00×Short 0.32×, long 0.73×Short 1.26×, long 0.86×Short 1.06×, long 1.01×Short 0.72×, long 0.19×Short 0.50×, long 0.07×Short 0.04×, long 0.01×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

About Sakana ALE-Bench: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by Sakana ALE-Bench's own grader, not a scorer AgentTime invented. It reports a score: Raw score on the private test cases, in each problem's own units; also reports rank against human contest entrants.

Tasks
10
Asked for
6 min 15 s to 28 h
Runs here
180
9 agents
Graded as
Raw score on the private test cases, in each problem's own units; also reports rank against human contest entrants.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 4 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Imajuku et al., 2025