Benchmark 2 of 18, 14 tasks, asked for 3 min 45 s to 24 h

DeepSWE

Original, hand-written software engineering tasks against real open-source repositories, such as fixing a label-sorting bug or reconstructing template strings in a policy engine's output.

By agent

Agents on DeepSWE ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6.1 SolCodex 86%24 of 28 runs 1.17× 85.714 graded
2 GPT 6 AstraCodex 58%14 of 24 runs 1.08× 83.312 graded
3 GPT 5.6 SolCodex 50%12 of 24 runs 1.76× 50.012 graded
4 Gemini 4 ArgonAntigravity CLI 4%1 of 28 runs 2.77× 92.914 graded
5 MiMo V2.6 ProMiMo Code 4%1 of 28 runs 6.26× 50.014 graded
6 Claude Opus 5.5Claude Code 0%0 of 28 runs 1.79× 92.914 graded
7 Claude Fable 5.1Claude Code 0%0 of 24 runs 3.49× 83.312 graded
8 Muse Wonder 1.0Muse Code 0%0 of 28 runs 9.05× 69.213 graded
9 MiMo V2.6 Pro UltraSpeedMiMo Code 0%0 of 28 runs 11.44× 50.014 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every DeepSWE run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

MiMo UltraSpeed's typical run lasted 0.24× the time asked at the short ask and 0.03× at the long one. Astra's lasted 1.10× and 1.01×.

Hover a mark to read it, click to pin it here.

Showing 240 runs.

The 14 tasks

The 14 tasks on DeepSWE, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
tomlkit-toml-table-converters4.5 min to 10 hNo runsNo runsNo runsShort 4.92×, long 0.89×Short 4.62×, long 0.93×Short 1.03×, long 1.01×Short 0.89×, long 0.07×Short 0.28×, long 0.02×Short 0.53×, long 0.03×
Fix PromQL label sorting across typed and untyped values5 min to 16 hShort 1.10×, long 1.01×Short 4.39×, long 1.02×Short 3.90×, long 0.36×Short 1.37×, long 0.80×Short 4.78×, long 0.69×Short 1.03×, long 1.01×Short 0.48×, long 0.04×Short 0.84×, long 0.05×Short 0.43×, long 0.11×
Fix isolated Go-side calls for Tengo callables and closures4.5 min to 16 hShort 1.21×, long 1.01×Short 2.89×, long 1.03×Short 7.26×, long 0.33×Short 0.75×, long 0.52×Short 7.94×, long 0.76×Short 1.02×, long 1.00×Short 0.47×, long 0.03×Short 0.21×, long 0.01×Short 0.57×, long 0.06×
Add RFC 5545 timezone interoperability to dateutil recurrence parsing4 min to 14 hShort 1.09×, long 1.02×Short 2.44×, long 1.01×Short 6.28×, long 0.46×Short 5.66×, long 0.76×Short 7.12×, long 0.50×Short 1.03×, long 1.01×Short 1.01×, long 0.06×Short 0.15×, long 0.02×Short 0.25×, long 0.03×
Add session bundle recording and replay to IPython3.75 min to 16 hShort 1.06×, long 1.01×Short 3.30×, long 1.02×Short 7.04×, long 0.52×Short 0.45×, long 0.48×Short 3.81×, long 0.90×Short 1.03×, long 1.00×Short 0.32×, long 0.02×Short 0.14×, long 0.01×Short 0.41×, long 0.02×
Add scoped state data to state-machine callbacks and history5 min to 24 hShort 1.46×, long 1.01×Short 3.94×, long 1.02×Short 5.50×, long 0.53×Short 0.69×, long 0.67×Short 4.73×, long 0.44×Short 1.02×, long 1.00×Short 0.39×, long 0.04×Short 0.14×, long 0.03×Short 0.39×, long 0.03×
Preserve structure needed by stylesheet selectors in OXVG8 min to 20 hShort 1.06×, long 1.01×Short 1.96×, long 1.01×Short 4.75×, long 0.37×Short 0.77×, long 0.79×Short 2.34×, long 0.80×Short 1.40×, long 1.00×Short 0.57×, long 0.06×Short 0.21×, long 0.04×Short 0.54×, long 0.10×
Add hierarchical evaluation cancellation to Boa8 min to 24 hShort 1.05×, long 1.02×Short 2.85×, long 1.01×Short 3.89×, long 0.32×Short 0.70×, long 0.93×Short 3.63×, long 0.32×Short 1.02×, long 0.16×Short 0.34×, long 0.03×Short 0.27×, long 0.02×Short 0.51×, long 0.05×
Add duration-aware sharding to Vitest5 min to 20 hShort 1.11×, long 1.02×Short 2.93×, long 1.01×Short 3.25×, long 0.35×Short 0.39×, long 0.58×Short 2.10×, long 0.24×Short 1.04×, long 1.01×Short 0.74×, long 0.05×Short 0.34×, long 0.02×Short 0.17×, long 0.02×
Add JSON Schema references and dependency keywords to ArkType5 min to 16 hShort 1.12×, long 1.01×Short 3.37×, long 1.01×Short 8.78×, long 0.40×Short 0.23×, long 0.84×Short 8.59×, long 1.01×Short 1.02×, long 0.37×Short 0.62×, long 0.04×Short 0.72×, long 0.08×Short 0.59×, long 0.03×
Add server-sent-event streaming endpoints to Effect HttpApi8 min to 14 hShort 1.44×, long 1.00×Short 2.01×, long 1.04×Short 1.28×, long 0.28×Short 0.33×, long 0.78×Short 7.13×, long 0.51×Short 2.65×, long 1.00×Short 1.07×, long 0.09×Short 0.52×, long 0.04×Short 0.75×, long 0.04×
Add multicolumn spans to KaTeX array-like environments4 min to 16 hShort 1.16×, long 1.01×Short 2.31×, long 1.01×Short 6.30×, long 0.39×Short 2.37×, long 0.86×Short 2.97×, long 0.77×Short 1.02×, long 1.00×Short 0.87×, long 0.05×Short 0.38×, long 0.03×Short 0.42×, long 0.04×
Add worktree merge conflict handling5.5 min to 18 hNo runsNo runsNo runsShort 2.29×, long 0.85×Short 6.27×, long 0.69×Short 1.02×, long 1.00×Short 0.35×, long 0.03×Short 0.20×, long 0.03×Short 0.23×, long 0.01×
Add deterministic map conflict detection to Y.Map writes5.5 min to 16 hNo runsNo runsNo runsShort 0.69×, long 0.35×Short 7.73×, long 0.88×Short 1.03×, long 1.00×Short 0.67×, long 0.05×Short 0.20×, long 0.04×Short 0.22×, long 0.01×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

The paper’s three agents also ran one task here that v1.1 replaced: Reconstruct template strings in OPA partial-evaluation output.

About DeepSWE: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by DeepSWE's own grader, not a scorer AgentTime invented. The pass rule: The hidden tests pass.

Tasks
14
Asked for
3 min 45 s to 24 h
Runs here
240
9 agents
Graded as
Pass or fail: The hidden tests pass.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 3 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Huang et al., 2026