Benchmark 3 of 18, 18 tasks, asked for 8 min to 280 h
ProgramBench
Given only a compiled binary and its documentation, rebuild the program's source code from scratch with no internet access, then pass its hidden tests.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6.1 SolCodex | 100%36 of 36 runs | 1.01× | 71.118 graded | |
| 2 | GPT 6 AstraCodex | 100%28 of 28 runs | 1.01× | 77.314 graded | |
| 3 | GPT 5.6 SolCodex | 29%8 of 28 runs | 2.16× | 72.614 graded | |
| 4 | Claude Opus 5.5Claude Code | 22%8 of 36 runs | 1.82× | 85.718 graded | |
| 5 | Gemini 4 ArgonAntigravity CLI | 6%2 of 36 runs | 2.91× | 59.118 graded | |
| 6 | Claude Fable 5.1Claude Code | 0%0 of 24 runs | 2.62× | 67.212 graded | |
| 7 | MiMo V2.6 ProMiMo Code | 0%0 of 36 runs | 10.12× | 60.117 graded | |
| 8 | MiMo V2.6 Pro UltraSpeedMiMo Code | 0%0 of 36 runs | 13.10× | 58.618 graded | |
| 9 | Muse Wonder 1.0Muse Code | 0%0 of 33 runs | 45.66× | 46.815 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every ProgramBench run
Muse Wonder's typical run lasted 0.10× the time asked at the short ask and 0.00× at the long one. Sol 6.1's lasted 1.02× and 1.00×.
Hover a mark to read it, click to pin it here.
Showing 293 runs.
The 18 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| cweill/gotests | 12.5 min to 40 h | Short 1.03×, long 1.00× | Short 1.58×, long 0.36× | Short 5.43×, long 0.40× | Short 0.39×, long 0.88× | Short 2.19×, long 0.47× | Short 1.03×, long 1.00× | Short 0.69×, long 0.06× | Short 0.49×, long 0.03× | Short 0.20×, long no run |
| nuta/nsh | 12.5 min to 48 h | Short 1.01×, long 1.00× | Short 1.77×, long 0.15× | Short 1.06×, long 2.00× | Short 0.78×, long 0.44× | Short 1.53×, long 0.33× | Short 1.02×, long 1.00× | Short 0.93×, long 0.10× | Short 0.65×, long 0.04× | Short 0.26×, long 0.01× |
| mookid/diffr | 11.25 min to 28 h | Short 1.01×, long 1.00× | Short 1.52×, long 1.01× | Short 33.33×, long 0.54× | Short 7.20×, long 0.50× | Short 9.85×, long 0.48× | Short 1.02×, long 1.00× | Short 1.66×, long 0.12× | Short 0.61×, long 0.04× | Short 0.31×, long 0.01× |
| ninja-build/ninja | 10 min to 160 h | Short 1.03×, long 1.00× | Short 2.39×, long 0.19× | Short 11.30×, long 0.49× | Short 2.08×, long 0.94× | Short 2.37×, long 0.26× | Short 1.03×, long 1.00× | Short 0.42×, long 0.04× | Short 0.12×, long 0.02× | Short 0.09×, long 0.00× |
| luajit/luajit | 12.5 min to 280 h | Short 1.01×, long 1.00× | Short 1.46×, long 1.00× | Short 1.42×, long 2.00× | Short 0.81×, long 1.75× | Short 1.95×, long 0.29× | Short 1.01×, long 1.00× | Short 0.26×, long 0.01× | Short 0.15×, long 0.01× | Short 0.02×, long 0.00× |
| facebook/zstd | 10 min to 160 h | Short 1.02×, long 1.01× | Short 1.78×, long 1.01× | Short 0.88×, long 0.40× | Short 0.97×, long 0.99× | Short 0.65×, long 0.14× | Short 1.04×, long 1.00× | Short 0.15×, long 0.01× | Short 0.10×, long 0.01× | Short 0.02×, long 0.00× |
| lfos/calcurse | 22.5 min to 60 h | Short 1.01×, long 1.00× | Short 0.95×, long 0.03× | Short 0.35×, long 0.81× | Short 0.22×, long 0.96× | Short 1.92×, long 0.26× | Short 1.02×, long 1.00× | Short 0.24×, long 0.03× | Short 0.60×, long 0.05× | Short 0.10×, long no run |
| tomarrell/wrapcheck | 10.5 min to 28 h | Short 1.03×, long 1.00× | Short 1.78×, long 1.01× | Short 7.04×, long 2.00× | Short 0.27×, long 0.47× | Short 4.26×, long 0.02× | Short 1.02×, long 1.00× | Short 0.88×, long 0.06× | Short 0.75×, long 0.03× | Short 0.26×, long 0.01× |
| doxygen/doxygen | 26.25 min to 200 h | No runs | No runs | No runs | Short 0.36×, long 0.93× | Short 1.09×, long 0.34× | Short 1.01×, long 1.00× | Short 0.22×, long 0.01× | Short 0.26×, long 0.02× | Short no run, long 0.03× |
| johnkerl/miller | 30 min to 160 h | No runs | No runs | No runs | Short 0.36×, long 0.48× | Short 1.55×, long 0.22× | Short 1.02×, long 1.00× | Short 0.38×, long 0.02× | Short 0.11×, long 0.03× | Short 0.39×, long 0.00× |
| paradigmxyz/solar | 18.75 min to 80 h | No runs | No runs | No runs | Short 0.28×, long 0.95× | Short 0.96×, long 0.91× | Short 1.02×, long 1.00× | Short 0.30×, long 0.01× | Short 0.25×, long 0.04× | Short 0.11×, long 0.00× |
| typst/typst | 26.25 min to 120 h | No runs | No runs | No runs | Short 0.29×, long 0.98× | Short 0.27×, long 0.19× | Short 1.01×, long 1.00× | Short 0.46×, long 0.43× | Short 0.24×, long 0.02× | Short 0.02×, long 0.00× |
| stranger6667/jsonschema | 15 min to 52 h | No runs | No runs | No runs | Short 0.79×, long 0.97× | Short 1.37×, long 0.30× | Short 1.03×, long 1.00× | Short 0.33×, long 0.04× | Short 0.48×, long 0.04× | Short 0.10×, long 0.01× |
| rvben/rumdl | 28.75 min to 100 h | No runs | No runs | No runs | Short 0.92×, long 0.99× | Short 1.62×, long 0.11× | Short 1.01×, long 1.00× | Short 0.17×, long 0.04× | Short 0.30×, long 0.02× | Short 0.05×, long 0.00× |
| jesseduffield/lazygit | 21.25 min to 160 h | No runs | No runs | No runs | Short 0.20×, long 2.00× | Short 1.03×, long 0.19× | Short 1.01×, long 1.00× | Short 0.13×, long 0.01× | Short 0.32×, long 0.02× | Short 0.15×, long 0.00× |
| dandavison/delta | 22.5 min to 180 h | No runs | No runs | No runs | Short 0.26×, long 0.92× | Short 3.07×, long 0.12× | Short 1.02×, long 1.00× | Short 0.78×, long 0.04× | Short 0.21×, long 0.01× | Short 0.06×, long 0.00× |
| canop/broot | 18.75 min to 168 h | No runs | No runs | No runs | Short 0.29×, long 0.65× | Short 1.06×, long 0.20× | Short 1.01×, long 1.00× | Short 0.18×, long 0.01× | Short 0.44×, long 0.01× | Short 0.09×, long 0.00× |
| ariga/atlas | 18.75 min to 120 h | No runs | No runs | No runs | Short 0.80×, long 0.95× | Short 0.91×, long 0.26× | Short 1.02×, long 1.01× | Short 0.16×, long 0.01× | Short 0.15×, long 0.02× | Short 0.05×, long 0.00× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
The paper’s three agents also ran 6 tasks here that v1.1 replaced: noborus/trdsql, rcoh/angle-grinder, stathissideris/ditaa, parcel-bundler/lightningcss, kaushiksrini/parqeye, o2sh/onefetch.
About ProgramBench: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by ProgramBench's own grader, not a scorer AgentTime invented. The pass rule: Resolved: every hidden test passes.
- Tasks
- 18
- Asked for
- 8 min to 280 h
- Runs here
- 293
- 9 agents
- Graded as
- Pass or fail: Resolved: every hidden test passes.
- Source
- Yang et al., 2026
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 13 runs here were stopped by AgentTime, so their time is a lower bound. Details
Data and citation
Upstream: Yang et al., 2026