Benchmark 4 of 18, 18 tasks, asked for 2 min 15 s to 10 h

OSWorld

Long, realistic desktop computer-use tasks, such as filing a reimbursement or booking a local event, done inside a real operating system.

By agent

Agents on OSWorld ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6 AstraCodex 53%17 of 32 runs 1.20× 67.316 graded
2 GPT 6.1 SolCodex 39%14 of 36 runs 1.62× 75.417 graded
3 Claude Fable 5.1Claude Code 19%6 of 32 runs 2.30× 76.216 graded
4 Claude Opus 5.5Claude Code 8%3 of 36 runs 2.10× 86.117 graded
5 MiMo V2.6 ProMiMo Code 8%3 of 36 runs 3.24× 61.918 graded
6 Gemini 4 ArgonAntigravity CLI 8%3 of 36 runs 4.04× 66.418 graded
7 MiMo V2.6 Pro UltraSpeedMiMo Code 6%2 of 36 runs 3.08× 64.418 graded
8 GPT 5.6 SolCodex 3%1 of 32 runs 4.48× 59.516 graded
9 Muse Wonder 1.0Muse Code 0%0 of 6 runs 4.25× 35.58 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every OSWorld run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Sol's typical run lasted 4.04× the time asked at the short ask and 0.14× at the long one. Astra's lasted 1.33× and 1.01×.

Hover a mark to read it, click to pin it here.

Showing 282 runs.

The 18 tasks

The 18 tasks on OSWorld, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
Defense schedule2.5 min to 4.17 hShort 1.10×, long 1.01×Short 4.14×, long 0.08×Short 2.01×, long 0.22×Short 2.62×, long 0.40×Short 6.36×, long 1.14×Short 1.02×, long 1.01×Short 30.58×, long 2.00×Short 22.90×, long 1.28×No runs
Reimbursement package3 min to 4.17 hShort 1.06×, long 1.00×Short 2.00×, long 0.04×Short 0.91×, long 0.19×Short 1.16×, long 0.23×Short 10.37×, long 1.02×Short 1.11×, long 1.01×Short 8.91×, long 0.03×Short 12.56×, long 0.44×Short 4.47×, long no run
Local event booking QA2.25 min to 4.17 hShort 1.24×, long 1.01×Short 1.59×, long 0.67×Short 0.83×, long 0.34×Short 0.97×, long 0.84×Short 1.27×, long 1.04×Short 1.15×, long 1.03×Short 1.37×, long 0.05×Short 3.81×, long 2.00×Short 11.77×, long no run
CRM leads2.25 min to 4.17 hShort 1.75×, long 1.01×Short 4.42×, long 0.11×Short 0.98×, long 0.30×Short 0.86×, long 0.37×Short 16.50×, long 1.08×Short 2.05×, long 1.02×Short 3.51×, long 1.77×Short 32.00×, long 2.00×No runs
Twenty recurring meetings7 min to 9.33 hShort 1.24×, long 1.00×Short 7.18×, long 0.18×Short 6.78×, long 0.21×Short 1.34×, long 0.15×Short 4.10×, long 0.31×Short 1.01×, long 0.04×Short 14.33×, long 1.05×Short 1.11×, long 1.13×Short 0.65×, long no run
Invoices, photos, and email5 min to 4.67 hShort 1.72×, long 1.02×Short 3.93×, long 0.10×Short 1.25×, long 0.16×Short 5.64×, long 0.93×Short 19.92×, long 1.51×Short 1.27×, long 1.02×Short 1.13×, long 0.98×Short 6.16×, long 1.31×No runs
GIMP composite3.5 min to 4.17 hShort 1.04×, long 1.00×Short 5.33×, long 0.09×Short 0.96×, long 0.52×Short 5.77×, long 0.75×Short 12.45×, long 1.03×Short 1.44×, long 0.33×Short 32.00×, long 1.52×Short 9.05×, long 0.96×No runs
GeoGebra vase and volume3.75 min to 4.17 hShort 1.59×, long 1.01×Short 2.35×, long 1.01×Short 1.02×, long 0.74×Short 1.17×, long 1.14×Short 6.41×, long 0.70×Short 1.18×, long 0.62×Short 4.52×, long 0.38×Short 1.84×, long 1.52×Short 1.30×, long 0.07×
MuseScore transcription5 min to 10 hShort 1.27×, long 1.01×Short 4.33×, long 0.18×Short 0.78×, long 0.63×Short 1.57×, long 1.13×Short 7.80×, long 0.54×Short 1.01×, long 0.71×Short 1.18×, long 0.61×Short 0.98×, long 0.71×No runs
Zotero, Obsidian, and BibTeX6.25 min to 6.33 hShort 1.35×, long 1.00×Short 5.79×, long 0.11×Short 0.80×, long 0.23×Short 2.56×, long 0.32×Short 1.89×, long 0.01×Short 1.04×, long 0.26×Short 1.03×, long 0.63×Short 3.76×, long 0.71×No runs
JSFX and REAPER render4.5 min to 7 hShort 1.88×, long 1.01×Short 2.26×, long 0.10×Short 34.00×, long 0.18×Short 4.09×, long 0.40×Short 3.26×, long 1.35×Short 1.41×, long 1.01×Short 3.44×, long 0.71×Short 2.05×, long 0.22×No runs
Radio bumper3.75 min to 4.17 hShort 1.50×, long 1.01×Short 5.08×, long 0.25×Short 1.81×, long 1.01×Short 3.99×, long 0.81×Short 19.05×, long 0.17×Short 1.09×, long 1.01×Short 9.51×, long 0.99×Short 4.31×, long 0.11×No runs
Calc macros, contracts, and emails5 min to 6 hShort 1.41×, long 1.01×Short 2.32×, long 0.21×Short 1.80×, long 0.35×Short 4.30×, long 0.76×Short 3.11×, long 0.43×Short 1.04×, long 0.25×Short 10.73×, long 1.82×Short 8.86×, long 1.09×No runs
Blender logo and Impress6.25 min to 4.17 hShort 3.04×, long 1.01×Short 3.36×, long 0.13×Short 5.63×, long 0.38×Short 1.01×, long 0.42×Short 1.31×, long 1.60×Short 1.08×, long 1.01×Short 8.13×, long 0.80×Short 19.98×, long 1.79×Short 3.85×, long no run
FreeCAD bracket6.5 min to 5 hShort 1.31×, long 1.01×Short 1.09×, long 0.14×Short 0.86×, long 2.04×Short 0.99×, long 1.32×Short 10.51×, long 1.07×Short 1.07×, long 0.19×Short 1.93×, long 0.93×Short 1.10×, long 0.60×No runs
KiCad schematic and PCB8 min to 10 hShort 1.28×, long 1.00×Short 8.26×, long 1.38×Short 0.95×, long 0.97×Short 6.01×, long 0.44×Short 28.05×, long 0.10×Short 1.33×, long 0.04×Short 4.59×, long 0.67×Short 5.21×, long 0.52×No runs
Young’s modulus report and fitted graph3.75 min to 5.67 hNo runsNo runsNo runsShort 0.92×, long 0.42×Short 10.68×, long 1.61×Short 1.29×, long 1.01×Short 6.46×, long 0.75×Short 5.97×, long 0.61×No runs
Shotcut montage project5.5 min to 8 hNo runsNo runsNo runsShort 1.66×, long 0.14×Short 28.70×, long 1.68×Short 1.12×, long 0.23×Short 13.54×, long 0.59×Short 10.65×, long 0.52×No runs

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

About OSWorld: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by OSWorld's own grader, not a scorer AgentTime invented. The pass rule: Binary completion: a score of 1.00, with every checkpoint met.

Tasks
18
Asked for
2 min 15 s to 10 h
Runs here
282
9 agents
Graded as
Pass or fail: Binary completion: a score of 1.00, with every checkpoint met.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 8 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Yuan et al., 2026