Benchmark 14 of 18, 10 tasks, asked for 6 min 15 s to 28 h
Sakana ALE-Bench
Competitive-programming optimization problems (drawn from AtCoder Heuristic Contests) with no known perfect answer, where the agent keeps improving its own solution.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreown units |
|---|---|---|---|---|---|
| 1 | GPT 6 AstraCodex | 95%19 of 20 runs | 1.02× | Own units | |
| 2 | GPT 6.1 SolCodex | 95%19 of 20 runs | 1.02× | Own units | |
| 3 | GPT 5.6 SolCodex | 65%13 of 20 runs | 1.16× | Own units | |
| 4 | Gemini 4 ArgonAntigravity CLI | 25%5 of 20 runs | 1.84× | Own units | |
| 5 | Claude Fable 5.1Claude Code | 10%2 of 20 runs | 1.95× | Own units | |
| 6 | MiMo V2.6 Pro UltraSpeedMiMo Code | 5%1 of 20 runs | 4.53× | Own units | |
| 7 | Claude Opus 5.5Claude Code | 0%0 of 20 runs | 1.86× | Own units | |
| 8 | MiMo V2.6 ProMiMo Code | 0%0 of 20 runs | 3.35× | Own units | |
| 9 | Muse Wonder 1.0Muse Code | 0%0 of 20 runs | 23.39× | Own units |
Sakana ALE-Bench keeps its own units, so it has no score out of 100. How it is measured
Every Sakana ALE-Bench run
Muse Wonder's typical run lasted 0.07× the time asked at the short ask and 0.01× at the long one. Astra's lasted 1.02× and 1.00×.
Hover a mark to read it, click to pin it here.
Showing 180 runs.
The 10 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| AtCoder Heuristic Contest 7 | 6.25 min to 15 h | Short 1.02×, long 1.00× | Short 1.29×, long 1.00× | Short 1.01×, long 0.42× | Short 0.89×, long 0.80× | Short 1.67×, long 0.96× | Short 1.03×, long 1.01× | Short 1.17×, long 0.19× | Short 1.04×, long 0.24× | Short 0.29×, long 0.01× |
| AtCoder Heuristic Contest 8 | 7.5 min to 28 h | Short 1.02×, long 1.00× | Short 1.05×, long 1.00× | Short 0.50×, long 0.27× | Short 0.40×, long 0.30× | Short 1.13×, long 0.11× | Short 1.02×, long 1.00× | Short 0.62×, long 0.07× | Short 0.19×, long 0.07× | Short 0.62×, long 0.03× |
| AtCoder Heuristic Contest 9 | 6.25 min to 16 h | Short 1.02×, long 1.00× | Short 1.10×, long 1.01× | Short 0.53×, long 0.55× | Short 0.37×, long 0.82× | Short 0.41×, long 1.00× | Short 1.04×, long 1.01× | Short 1.16×, long 0.21× | Short 1.19×, long 0.05× | Short 0.07×, long 0.01× |
| AtCoder Heuristic Contest 11 | 11.25 min to 25.3 h | Short 1.02×, long 1.00× | Short 1.03×, long 1.00× | Short 0.99×, long 0.30× | Short 0.75×, long 0.79× | Short 1.08×, long 0.41× | Short 1.04×, long 1.00× | Short 0.74×, long 0.10× | Short 0.38×, long 0.16× | Short 0.07×, long 0.01× |
| Future Contest 2022 Qual | 10 min to 24 h | Short 1.03×, long 1.00× | Short 1.03×, long 1.01× | Short 0.60×, long 0.28× | Short 0.83×, long 0.55× | Short 1.40×, long 1.00× | Short 1.02×, long 1.00× | Short 1.83×, long 0.14× | Short 0.58×, long 0.12× | Short 0.09×, long 0.04× |
| AtCoder Heuristic Contest 16 | 10 min to 18 h | Short 1.02×, long 1.01× | Short 1.81×, long 1.00× | Short 0.36×, long 0.40× | Short 0.24×, long 0.33× | Short 1.11×, long 0.83× | Short 1.04×, long 1.00× | Short 0.77×, long 0.14× | Short 0.79×, long 0.06× | Short 0.88×, long 0.06× |
| AtCoder Heuristic Contest 26 | 6.25 min to 20 h | Short 1.02×, long 1.01× | Short 1.53×, long 1.00× | Short 0.78×, long 0.48× | Short 0.41×, long 0.37× | Short 1.02×, long 1.01× | Short 1.03×, long 1.01× | Short 0.73×, long 0.09× | Short 0.81×, long 0.04× | Short 0.07×, long 0.04× |
| AtCoder Heuristic Contest 31 | 10 min to 20 h | Short 1.11×, long 1.00× | Short 1.09×, long 1.05× | Short 0.67×, long 0.51× | Short 0.39×, long 0.84× | Short 1.11×, long 0.12× | Short 1.03×, long 1.00× | Short 0.68×, long 0.12× | Short 0.35×, long 0.14× | Short 0.03×, long 0.01× |
| AtCoder Heuristic Contest 39 | 8.75 min to 20 h | Short 1.04×, long 1.00× | Short 1.24×, long 1.02× | Short 0.60×, long 0.49× | Short 0.75×, long 0.83× | Short 1.15×, long 0.02× | Short 1.03×, long 1.00× | Short 0.76×, long 0.06× | Short 0.25×, long 0.19× | Short 0.06×, long 0.01× |
| AtCoder Heuristic Contest 46 | 7.5 min to 11 h | Short 1.01×, long 1.00× | Short 2.89×, long 1.01× | Short 0.70×, long 2.00× | Short 0.32×, long 0.73× | Short 1.26×, long 0.86× | Short 1.06×, long 1.01× | Short 0.72×, long 0.19× | Short 0.50×, long 0.07× | Short 0.04×, long 0.01× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
About Sakana ALE-Bench: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by Sakana ALE-Bench's own grader, not a scorer AgentTime invented. It reports a score: Raw score on the private test cases, in each problem's own units; also reports rank against human contest entrants.
- Tasks
- 10
- Asked for
- 6 min 15 s to 28 h
- Runs here
- 180
- 9 agents
- Graded as
- Raw score on the private test cases, in each problem's own units; also reports rank against human contest entrants.
- Source
- Imajuku et al., 2025
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 4 runs here were stopped by AgentTime, so their time is a lower bound. Details
Data and citation
Upstream: Imajuku et al., 2025