Benchmark 2 of 18, 14 tasks, asked for 3 min 45 s to 24 h
DeepSWE
Original, hand-written software engineering tasks against real open-source repositories, such as fixing a label-sorting bug or reconstructing template strings in a policy engine's output.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6.1 SolCodex | 86%24 of 28 runs | 1.17× | 85.714 graded | |
| 2 | GPT 6 AstraCodex | 58%14 of 24 runs | 1.08× | 83.312 graded | |
| 3 | GPT 5.6 SolCodex | 50%12 of 24 runs | 1.76× | 50.012 graded | |
| 4 | Gemini 4 ArgonAntigravity CLI | 4%1 of 28 runs | 2.77× | 92.914 graded | |
| 5 | MiMo V2.6 ProMiMo Code | 4%1 of 28 runs | 6.26× | 50.014 graded | |
| 6 | Claude Opus 5.5Claude Code | 0%0 of 28 runs | 1.79× | 92.914 graded | |
| 7 | Claude Fable 5.1Claude Code | 0%0 of 24 runs | 3.49× | 83.312 graded | |
| 8 | Muse Wonder 1.0Muse Code | 0%0 of 28 runs | 9.05× | 69.213 graded | |
| 9 | MiMo V2.6 Pro UltraSpeedMiMo Code | 0%0 of 28 runs | 11.44× | 50.014 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every DeepSWE run
MiMo UltraSpeed's typical run lasted 0.24× the time asked at the short ask and 0.03× at the long one. Astra's lasted 1.10× and 1.01×.
Hover a mark to read it, click to pin it here.
Showing 240 runs.
The 14 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| tomlkit-toml-table-converters | 4.5 min to 10 h | No runs | No runs | No runs | Short 4.92×, long 0.89× | Short 4.62×, long 0.93× | Short 1.03×, long 1.01× | Short 0.89×, long 0.07× | Short 0.28×, long 0.02× | Short 0.53×, long 0.03× |
| Fix PromQL label sorting across typed and untyped values | 5 min to 16 h | Short 1.10×, long 1.01× | Short 4.39×, long 1.02× | Short 3.90×, long 0.36× | Short 1.37×, long 0.80× | Short 4.78×, long 0.69× | Short 1.03×, long 1.01× | Short 0.48×, long 0.04× | Short 0.84×, long 0.05× | Short 0.43×, long 0.11× |
| Fix isolated Go-side calls for Tengo callables and closures | 4.5 min to 16 h | Short 1.21×, long 1.01× | Short 2.89×, long 1.03× | Short 7.26×, long 0.33× | Short 0.75×, long 0.52× | Short 7.94×, long 0.76× | Short 1.02×, long 1.00× | Short 0.47×, long 0.03× | Short 0.21×, long 0.01× | Short 0.57×, long 0.06× |
| Add RFC 5545 timezone interoperability to dateutil recurrence parsing | 4 min to 14 h | Short 1.09×, long 1.02× | Short 2.44×, long 1.01× | Short 6.28×, long 0.46× | Short 5.66×, long 0.76× | Short 7.12×, long 0.50× | Short 1.03×, long 1.01× | Short 1.01×, long 0.06× | Short 0.15×, long 0.02× | Short 0.25×, long 0.03× |
| Add session bundle recording and replay to IPython | 3.75 min to 16 h | Short 1.06×, long 1.01× | Short 3.30×, long 1.02× | Short 7.04×, long 0.52× | Short 0.45×, long 0.48× | Short 3.81×, long 0.90× | Short 1.03×, long 1.00× | Short 0.32×, long 0.02× | Short 0.14×, long 0.01× | Short 0.41×, long 0.02× |
| Add scoped state data to state-machine callbacks and history | 5 min to 24 h | Short 1.46×, long 1.01× | Short 3.94×, long 1.02× | Short 5.50×, long 0.53× | Short 0.69×, long 0.67× | Short 4.73×, long 0.44× | Short 1.02×, long 1.00× | Short 0.39×, long 0.04× | Short 0.14×, long 0.03× | Short 0.39×, long 0.03× |
| Preserve structure needed by stylesheet selectors in OXVG | 8 min to 20 h | Short 1.06×, long 1.01× | Short 1.96×, long 1.01× | Short 4.75×, long 0.37× | Short 0.77×, long 0.79× | Short 2.34×, long 0.80× | Short 1.40×, long 1.00× | Short 0.57×, long 0.06× | Short 0.21×, long 0.04× | Short 0.54×, long 0.10× |
| Add hierarchical evaluation cancellation to Boa | 8 min to 24 h | Short 1.05×, long 1.02× | Short 2.85×, long 1.01× | Short 3.89×, long 0.32× | Short 0.70×, long 0.93× | Short 3.63×, long 0.32× | Short 1.02×, long 0.16× | Short 0.34×, long 0.03× | Short 0.27×, long 0.02× | Short 0.51×, long 0.05× |
| Add duration-aware sharding to Vitest | 5 min to 20 h | Short 1.11×, long 1.02× | Short 2.93×, long 1.01× | Short 3.25×, long 0.35× | Short 0.39×, long 0.58× | Short 2.10×, long 0.24× | Short 1.04×, long 1.01× | Short 0.74×, long 0.05× | Short 0.34×, long 0.02× | Short 0.17×, long 0.02× |
| Add JSON Schema references and dependency keywords to ArkType | 5 min to 16 h | Short 1.12×, long 1.01× | Short 3.37×, long 1.01× | Short 8.78×, long 0.40× | Short 0.23×, long 0.84× | Short 8.59×, long 1.01× | Short 1.02×, long 0.37× | Short 0.62×, long 0.04× | Short 0.72×, long 0.08× | Short 0.59×, long 0.03× |
| Add server-sent-event streaming endpoints to Effect HttpApi | 8 min to 14 h | Short 1.44×, long 1.00× | Short 2.01×, long 1.04× | Short 1.28×, long 0.28× | Short 0.33×, long 0.78× | Short 7.13×, long 0.51× | Short 2.65×, long 1.00× | Short 1.07×, long 0.09× | Short 0.52×, long 0.04× | Short 0.75×, long 0.04× |
| Add multicolumn spans to KaTeX array-like environments | 4 min to 16 h | Short 1.16×, long 1.01× | Short 2.31×, long 1.01× | Short 6.30×, long 0.39× | Short 2.37×, long 0.86× | Short 2.97×, long 0.77× | Short 1.02×, long 1.00× | Short 0.87×, long 0.05× | Short 0.38×, long 0.03× | Short 0.42×, long 0.04× |
| Add worktree merge conflict handling | 5.5 min to 18 h | No runs | No runs | No runs | Short 2.29×, long 0.85× | Short 6.27×, long 0.69× | Short 1.02×, long 1.00× | Short 0.35×, long 0.03× | Short 0.20×, long 0.03× | Short 0.23×, long 0.01× |
| Add deterministic map conflict detection to Y.Map writes | 5.5 min to 16 h | No runs | No runs | No runs | Short 0.69×, long 0.35× | Short 7.73×, long 0.88× | Short 1.03×, long 1.00× | Short 0.67×, long 0.05× | Short 0.20×, long 0.04× | Short 0.22×, long 0.01× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
The paper’s three agents also ran one task here that v1.1 replaced: Reconstruct template strings in OPA partial-evaluation output.
About DeepSWE: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by DeepSWE's own grader, not a scorer AgentTime invented. The pass rule: The hidden tests pass.
- Tasks
- 14
- Asked for
- 3 min 45 s to 24 h
- Runs here
- 240
- 9 agents
- Graded as
- Pass or fail: The hidden tests pass.
- Source
- Huang et al., 2026
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 3 runs here were stopped by AgentTime, so their time is a lower bound. Details
Data and citation
Upstream: Huang et al., 2026