Benchmark 3 of 18, 18 tasks, asked for 8 min to 280 h

ProgramBench

Given only a compiled binary and its documentation, rebuild the program's source code from scratch with no internet access, then pass its hidden tests.

By agent

Agents on ProgramBench ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6.1 SolCodex 100%36 of 36 runs 1.01× 71.118 graded
2 GPT 6 AstraCodex 100%28 of 28 runs 1.01× 77.314 graded
3 GPT 5.6 SolCodex 29%8 of 28 runs 2.16× 72.614 graded
4 Claude Opus 5.5Claude Code 22%8 of 36 runs 1.82× 85.718 graded
5 Gemini 4 ArgonAntigravity CLI 6%2 of 36 runs 2.91× 59.118 graded
6 Claude Fable 5.1Claude Code 0%0 of 24 runs 2.62× 67.212 graded
7 MiMo V2.6 ProMiMo Code 0%0 of 36 runs 10.12× 60.117 graded
8 MiMo V2.6 Pro UltraSpeedMiMo Code 0%0 of 36 runs 13.10× 58.618 graded
9 Muse Wonder 1.0Muse Code 0%0 of 33 runs 45.66× 46.815 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every ProgramBench run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Muse Wonder's typical run lasted 0.10× the time asked at the short ask and 0.00× at the long one. Sol 6.1's lasted 1.02× and 1.00×.

Hover a mark to read it, click to pin it here.

Showing 293 runs.

The 18 tasks

The 18 tasks on ProgramBench, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
cweill/gotests12.5 min to 40 hShort 1.03×, long 1.00×Short 1.58×, long 0.36×Short 5.43×, long 0.40×Short 0.39×, long 0.88×Short 2.19×, long 0.47×Short 1.03×, long 1.00×Short 0.69×, long 0.06×Short 0.49×, long 0.03×Short 0.20×, long no run
nuta/nsh12.5 min to 48 hShort 1.01×, long 1.00×Short 1.77×, long 0.15×Short 1.06×, long 2.00×Short 0.78×, long 0.44×Short 1.53×, long 0.33×Short 1.02×, long 1.00×Short 0.93×, long 0.10×Short 0.65×, long 0.04×Short 0.26×, long 0.01×
mookid/diffr11.25 min to 28 hShort 1.01×, long 1.00×Short 1.52×, long 1.01×Short 33.33×, long 0.54×Short 7.20×, long 0.50×Short 9.85×, long 0.48×Short 1.02×, long 1.00×Short 1.66×, long 0.12×Short 0.61×, long 0.04×Short 0.31×, long 0.01×
ninja-build/ninja10 min to 160 hShort 1.03×, long 1.00×Short 2.39×, long 0.19×Short 11.30×, long 0.49×Short 2.08×, long 0.94×Short 2.37×, long 0.26×Short 1.03×, long 1.00×Short 0.42×, long 0.04×Short 0.12×, long 0.02×Short 0.09×, long 0.00×
luajit/luajit12.5 min to 280 hShort 1.01×, long 1.00×Short 1.46×, long 1.00×Short 1.42×, long 2.00×Short 0.81×, long 1.75×Short 1.95×, long 0.29×Short 1.01×, long 1.00×Short 0.26×, long 0.01×Short 0.15×, long 0.01×Short 0.02×, long 0.00×
facebook/zstd10 min to 160 hShort 1.02×, long 1.01×Short 1.78×, long 1.01×Short 0.88×, long 0.40×Short 0.97×, long 0.99×Short 0.65×, long 0.14×Short 1.04×, long 1.00×Short 0.15×, long 0.01×Short 0.10×, long 0.01×Short 0.02×, long 0.00×
lfos/calcurse22.5 min to 60 hShort 1.01×, long 1.00×Short 0.95×, long 0.03×Short 0.35×, long 0.81×Short 0.22×, long 0.96×Short 1.92×, long 0.26×Short 1.02×, long 1.00×Short 0.24×, long 0.03×Short 0.60×, long 0.05×Short 0.10×, long no run
tomarrell/wrapcheck10.5 min to 28 hShort 1.03×, long 1.00×Short 1.78×, long 1.01×Short 7.04×, long 2.00×Short 0.27×, long 0.47×Short 4.26×, long 0.02×Short 1.02×, long 1.00×Short 0.88×, long 0.06×Short 0.75×, long 0.03×Short 0.26×, long 0.01×
doxygen/doxygen26.25 min to 200 hNo runsNo runsNo runsShort 0.36×, long 0.93×Short 1.09×, long 0.34×Short 1.01×, long 1.00×Short 0.22×, long 0.01×Short 0.26×, long 0.02×Short no run, long 0.03×
johnkerl/miller30 min to 160 hNo runsNo runsNo runsShort 0.36×, long 0.48×Short 1.55×, long 0.22×Short 1.02×, long 1.00×Short 0.38×, long 0.02×Short 0.11×, long 0.03×Short 0.39×, long 0.00×
paradigmxyz/solar18.75 min to 80 hNo runsNo runsNo runsShort 0.28×, long 0.95×Short 0.96×, long 0.91×Short 1.02×, long 1.00×Short 0.30×, long 0.01×Short 0.25×, long 0.04×Short 0.11×, long 0.00×
typst/typst26.25 min to 120 hNo runsNo runsNo runsShort 0.29×, long 0.98×Short 0.27×, long 0.19×Short 1.01×, long 1.00×Short 0.46×, long 0.43×Short 0.24×, long 0.02×Short 0.02×, long 0.00×
stranger6667/jsonschema15 min to 52 hNo runsNo runsNo runsShort 0.79×, long 0.97×Short 1.37×, long 0.30×Short 1.03×, long 1.00×Short 0.33×, long 0.04×Short 0.48×, long 0.04×Short 0.10×, long 0.01×
rvben/rumdl28.75 min to 100 hNo runsNo runsNo runsShort 0.92×, long 0.99×Short 1.62×, long 0.11×Short 1.01×, long 1.00×Short 0.17×, long 0.04×Short 0.30×, long 0.02×Short 0.05×, long 0.00×
jesseduffield/lazygit21.25 min to 160 hNo runsNo runsNo runsShort 0.20×, long 2.00×Short 1.03×, long 0.19×Short 1.01×, long 1.00×Short 0.13×, long 0.01×Short 0.32×, long 0.02×Short 0.15×, long 0.00×
dandavison/delta22.5 min to 180 hNo runsNo runsNo runsShort 0.26×, long 0.92×Short 3.07×, long 0.12×Short 1.02×, long 1.00×Short 0.78×, long 0.04×Short 0.21×, long 0.01×Short 0.06×, long 0.00×
canop/broot18.75 min to 168 hNo runsNo runsNo runsShort 0.29×, long 0.65×Short 1.06×, long 0.20×Short 1.01×, long 1.00×Short 0.18×, long 0.01×Short 0.44×, long 0.01×Short 0.09×, long 0.00×
ariga/atlas18.75 min to 120 hNo runsNo runsNo runsShort 0.80×, long 0.95×Short 0.91×, long 0.26×Short 1.02×, long 1.01×Short 0.16×, long 0.01×Short 0.15×, long 0.02×Short 0.05×, long 0.00×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

The paper’s three agents also ran 6 tasks here that v1.1 replaced: noborus/trdsql, rcoh/angle-grinder, stathissideris/ditaa, parcel-bundler/lightningcss, kaushiksrini/parqeye, o2sh/onefetch.

About ProgramBench: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by ProgramBench's own grader, not a scorer AgentTime invented. The pass rule: Resolved: every hidden test passes.

Tasks
18
Asked for
8 min to 280 h
Runs here
293
9 agents
Graded as
Pass or fail: Resolved: every hidden test passes.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 13 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Yang et al., 2026