Benchmark 4 of 18, 18 tasks, asked for 2 min 15 s to 10 h
OSWorld
Long, realistic desktop computer-use tasks, such as filing a reimbursement or booking a local event, done inside a real operating system.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6 AstraCodex | 53%17 of 32 runs | 1.20× | 67.316 graded | |
| 2 | GPT 6.1 SolCodex | 39%14 of 36 runs | 1.62× | 75.417 graded | |
| 3 | Claude Fable 5.1Claude Code | 19%6 of 32 runs | 2.30× | 76.216 graded | |
| 4 | Claude Opus 5.5Claude Code | 8%3 of 36 runs | 2.10× | 86.117 graded | |
| 5 | MiMo V2.6 ProMiMo Code | 8%3 of 36 runs | 3.24× | 61.918 graded | |
| 6 | Gemini 4 ArgonAntigravity CLI | 8%3 of 36 runs | 4.04× | 66.418 graded | |
| 7 | MiMo V2.6 Pro UltraSpeedMiMo Code | 6%2 of 36 runs | 3.08× | 64.418 graded | |
| 8 | GPT 5.6 SolCodex | 3%1 of 32 runs | 4.48× | 59.516 graded | |
| 9 | Muse Wonder 1.0Muse Code | 0%0 of 6 runs | 4.25× | 35.58 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every OSWorld run
Sol's typical run lasted 4.04× the time asked at the short ask and 0.14× at the long one. Astra's lasted 1.33× and 1.01×.
Hover a mark to read it, click to pin it here.
Showing 282 runs.
The 18 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| Defense schedule | 2.5 min to 4.17 h | Short 1.10×, long 1.01× | Short 4.14×, long 0.08× | Short 2.01×, long 0.22× | Short 2.62×, long 0.40× | Short 6.36×, long 1.14× | Short 1.02×, long 1.01× | Short 30.58×, long 2.00× | Short 22.90×, long 1.28× | No runs |
| Reimbursement package | 3 min to 4.17 h | Short 1.06×, long 1.00× | Short 2.00×, long 0.04× | Short 0.91×, long 0.19× | Short 1.16×, long 0.23× | Short 10.37×, long 1.02× | Short 1.11×, long 1.01× | Short 8.91×, long 0.03× | Short 12.56×, long 0.44× | Short 4.47×, long no run |
| Local event booking QA | 2.25 min to 4.17 h | Short 1.24×, long 1.01× | Short 1.59×, long 0.67× | Short 0.83×, long 0.34× | Short 0.97×, long 0.84× | Short 1.27×, long 1.04× | Short 1.15×, long 1.03× | Short 1.37×, long 0.05× | Short 3.81×, long 2.00× | Short 11.77×, long no run |
| CRM leads | 2.25 min to 4.17 h | Short 1.75×, long 1.01× | Short 4.42×, long 0.11× | Short 0.98×, long 0.30× | Short 0.86×, long 0.37× | Short 16.50×, long 1.08× | Short 2.05×, long 1.02× | Short 3.51×, long 1.77× | Short 32.00×, long 2.00× | No runs |
| Twenty recurring meetings | 7 min to 9.33 h | Short 1.24×, long 1.00× | Short 7.18×, long 0.18× | Short 6.78×, long 0.21× | Short 1.34×, long 0.15× | Short 4.10×, long 0.31× | Short 1.01×, long 0.04× | Short 14.33×, long 1.05× | Short 1.11×, long 1.13× | Short 0.65×, long no run |
| Invoices, photos, and email | 5 min to 4.67 h | Short 1.72×, long 1.02× | Short 3.93×, long 0.10× | Short 1.25×, long 0.16× | Short 5.64×, long 0.93× | Short 19.92×, long 1.51× | Short 1.27×, long 1.02× | Short 1.13×, long 0.98× | Short 6.16×, long 1.31× | No runs |
| GIMP composite | 3.5 min to 4.17 h | Short 1.04×, long 1.00× | Short 5.33×, long 0.09× | Short 0.96×, long 0.52× | Short 5.77×, long 0.75× | Short 12.45×, long 1.03× | Short 1.44×, long 0.33× | Short 32.00×, long 1.52× | Short 9.05×, long 0.96× | No runs |
| GeoGebra vase and volume | 3.75 min to 4.17 h | Short 1.59×, long 1.01× | Short 2.35×, long 1.01× | Short 1.02×, long 0.74× | Short 1.17×, long 1.14× | Short 6.41×, long 0.70× | Short 1.18×, long 0.62× | Short 4.52×, long 0.38× | Short 1.84×, long 1.52× | Short 1.30×, long 0.07× |
| MuseScore transcription | 5 min to 10 h | Short 1.27×, long 1.01× | Short 4.33×, long 0.18× | Short 0.78×, long 0.63× | Short 1.57×, long 1.13× | Short 7.80×, long 0.54× | Short 1.01×, long 0.71× | Short 1.18×, long 0.61× | Short 0.98×, long 0.71× | No runs |
| Zotero, Obsidian, and BibTeX | 6.25 min to 6.33 h | Short 1.35×, long 1.00× | Short 5.79×, long 0.11× | Short 0.80×, long 0.23× | Short 2.56×, long 0.32× | Short 1.89×, long 0.01× | Short 1.04×, long 0.26× | Short 1.03×, long 0.63× | Short 3.76×, long 0.71× | No runs |
| JSFX and REAPER render | 4.5 min to 7 h | Short 1.88×, long 1.01× | Short 2.26×, long 0.10× | Short 34.00×, long 0.18× | Short 4.09×, long 0.40× | Short 3.26×, long 1.35× | Short 1.41×, long 1.01× | Short 3.44×, long 0.71× | Short 2.05×, long 0.22× | No runs |
| Radio bumper | 3.75 min to 4.17 h | Short 1.50×, long 1.01× | Short 5.08×, long 0.25× | Short 1.81×, long 1.01× | Short 3.99×, long 0.81× | Short 19.05×, long 0.17× | Short 1.09×, long 1.01× | Short 9.51×, long 0.99× | Short 4.31×, long 0.11× | No runs |
| Calc macros, contracts, and emails | 5 min to 6 h | Short 1.41×, long 1.01× | Short 2.32×, long 0.21× | Short 1.80×, long 0.35× | Short 4.30×, long 0.76× | Short 3.11×, long 0.43× | Short 1.04×, long 0.25× | Short 10.73×, long 1.82× | Short 8.86×, long 1.09× | No runs |
| Blender logo and Impress | 6.25 min to 4.17 h | Short 3.04×, long 1.01× | Short 3.36×, long 0.13× | Short 5.63×, long 0.38× | Short 1.01×, long 0.42× | Short 1.31×, long 1.60× | Short 1.08×, long 1.01× | Short 8.13×, long 0.80× | Short 19.98×, long 1.79× | Short 3.85×, long no run |
| FreeCAD bracket | 6.5 min to 5 h | Short 1.31×, long 1.01× | Short 1.09×, long 0.14× | Short 0.86×, long 2.04× | Short 0.99×, long 1.32× | Short 10.51×, long 1.07× | Short 1.07×, long 0.19× | Short 1.93×, long 0.93× | Short 1.10×, long 0.60× | No runs |
| KiCad schematic and PCB | 8 min to 10 h | Short 1.28×, long 1.00× | Short 8.26×, long 1.38× | Short 0.95×, long 0.97× | Short 6.01×, long 0.44× | Short 28.05×, long 0.10× | Short 1.33×, long 0.04× | Short 4.59×, long 0.67× | Short 5.21×, long 0.52× | No runs |
| Young’s modulus report and fitted graph | 3.75 min to 5.67 h | No runs | No runs | No runs | Short 0.92×, long 0.42× | Short 10.68×, long 1.61× | Short 1.29×, long 1.01× | Short 6.46×, long 0.75× | Short 5.97×, long 0.61× | No runs |
| Shotcut montage project | 5.5 min to 8 h | No runs | No runs | No runs | Short 1.66×, long 0.14× | Short 28.70×, long 1.68× | Short 1.12×, long 0.23× | Short 13.54×, long 0.59× | Short 10.65×, long 0.52× | No runs |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
About OSWorld: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by OSWorld's own grader, not a scorer AgentTime invented. The pass rule: Binary completion: a score of 1.00, with every checkpoint met.
- Tasks
- 18
- Asked for
- 2 min 15 s to 10 h
- Runs here
- 282
- 9 agents
- Graded as
- Pass or fail: Binary completion: a score of 1.00, with every checkpoint met.
- Source
- Yuan et al., 2026
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 8 runs here were stopped by AgentTime, so their time is a lower bound. Details
Data and citation
Upstream: Yuan et al., 2026