Benchmark 5 of 18, 14 tasks, asked for 23 s to 5 h

AppWorld

Simulated everyday app tasks, like exporting a playlist to a spreadsheet or reconciling an emailed song list with what is actually on Spotify, done by calling app APIs.

By agent

Agents on AppWorld ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6 AstraCodex 50%18 of 36 runs 1.09× 66.718 graded
2 GPT 6.1 SolCodex 50%14 of 28 runs 1.09× 57.114 graded
3 GPT 5.6 SolCodex 36%13 of 36 runs 1.15× 66.718 graded
4 MiMo V2.6 ProMiMo Code 11%3 of 28 runs 3.61× 35.714 graded
5 Gemini 4 ArgonAntigravity CLI 7%2 of 28 runs 3.95× 57.114 graded
6 Muse Wonder 1.0Muse Code 7%2 of 28 runs 11.50× 50.014 graded
7 MiMo V2.6 Pro UltraSpeedMiMo Code 4%1 of 28 runs 5.97× 35.714 graded
8 Claude Fable 5.1Claude Code 0%0 of 36 runs 3.63× 72.218 graded
9 Claude Opus 5.5Claude Code 0%0 of 28 runs 4.02× 57.114 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every AppWorld run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Muse Wonder's typical run lasted 0.79× the time asked at the short ask and 0.02× at the long one. Astra's lasted 1.15× and 1.03×.

Hover a mark to read it, click to pin it here.

Showing 276 runs.

The 14 tasks

The 14 tasks on AppWorld, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
Execute every task in a camping project across shopping, expenses, payments, invitations, and task closeout1.5 min to 5 hShort 1.08×, long 1.02×Short 1.33×, long 1.02×Short 2.74×, long 0.19×Short 1.88×, long 0.22×Short 32.00×, long 2.00×Short 1.09×, long 1.01×Short 2.37×, long 1.00×Short 1.55×, long 0.12×Short 0.15×, long 0.04×
Reconcile an emailed road-trip song list with a Spotify playlist and download it22.5 s to 1.2 hShort 1.12×, long 1.03×Short 2.12×, long 1.05×Short 1.92×, long 0.14×Short 0.45×, long 0.05×Short 12.29×, long 0.89×Short 1.21×, long 1.02×Short 0.99×, long 1.10×Short 2.03×, long 0.05×Short 1.30×, long 0.04×
Turn this year’s emailed electricity bills into monthly Splitwise expenses37.5 s to 3 hShort 1.16×, long 1.02×Short 1.59×, long 1.03×Short 1.96×, long 0.14×Short 0.58×, long 0.09×Short 7.10×, long 0.96×Short 1.17×, long 1.02×Short 0.62×, long 0.04×Short 0.47×, long 0.03×Short 1.03×, long 0.03×
Buy a requested item, forward its receipt, and recover the cost through Venmo37.5 s to 1.87 hShort 1.24×, long 1.04×Short 1.13×, long 1.05×Short 2.22×, long 0.16×Short 2.06×, long 0.22×Short 32.00×, long 2.00×Short 1.15×, long 1.03×Short 4.40×, long 0.53×Short 1.18×, long 0.17×Short 0.04×, long 0.00×
Collect artist recommendations from email and text conversations, then follow them30 s to 1.87 hShort 1.11×, long 1.02×Short 1.15×, long 1.03×Short 1.46×, long 0.12×Short 0.72×, long 0.05×Short 9.84×, long 0.89×Short 1.13×, long 1.02×Short 0.75×, long 0.04×Short 1.18×, long 0.06×Short 0.41×, long 0.01×
Rebuild and send a drafted recommendation email from the user’s Spotify history37.5 s to 1.67 hShort 1.20×, long 1.04×Short 1.24×, long 1.05×Short 1.30×, long 0.15×Short 0.62×, long 0.06×Short 5.94×, long 0.56×Short 1.12×, long 1.02×Short 1.05×, long 0.07×Short 0.26×, long 0.02×Short 0.88×, long 0.03×
Merge two photo collections into a ZIP and send it to both parents30 s to 1.67 hShort 1.09×, long 1.02×Short 1.17×, long 1.03×Short 1.75×, long 0.14×Short 0.83×, long 0.09×Short 7.14×, long 1.17×Short 1.21×, long 1.03×Short 1.26×, long 0.09×Short 0.23×, long 0.02×Short 1.28×, long 0.04×
Reconcile a camping checklist with commitments and buy the remaining supplies37.5 s to 3 hShort 1.14×, long 1.04×Short 1.15×, long 1.03×Short 1.94×, long 0.17×Short 2.11×, long 0.33×Short 32.00×, long 2.00×Short 1.14×, long 1.03×Short 2.88×, long 1.00×Short 1.53×, long 0.15×Short 1.12×, long 0.06×
Collect every cable bill, normalize filenames, ZIP them, and reply to the request37.5 s to 2.8 hShort 1.08×, long 1.03×Short 1.19×, long 1.02×Short 2.04×, long 0.12×Short 1.07×, long 0.09×Short 6.10×, long 1.43×Short 1.14×, long 1.03×Short 0.44×, long 0.13×Short 1.09×, long 0.09×Short 0.05×, long 0.01×
Turn a note-based meeting schedule into correctly timed reminder emails30 s to 2.33 hShort 1.16×, long 1.03×Short 0.84×, long 1.07×Short 2.11×, long 0.14×Short 0.81×, long 0.12×Short 3.62×, long 1.06×Short 1.24×, long 1.02×Short 0.70×, long 0.03×Short 0.38×, long 0.03×Short 1.05×, long 0.02×
Build a recruiting CSV and archive each candidate’s attachments37.5 s to 2 hShort 1.22×, long 1.03×Short 1.44×, long 1.04×Short 1.65×, long 0.39×Short 0.67×, long 0.08×Short 22.24×, long 2.00×Short 1.17×, long 1.02×Short 0.58×, long 0.07×Short 0.79×, long 0.09×Short 0.21×, long 0.02×
Repack tonight’s Todoist plan around the available work schedule45 s to 2.13 hShort 1.21×, long 1.04×Short 1.48×, long 1.04×Short 1.91×, long 0.17×Short 0.81×, long 0.22×Short 13.11×, long 1.05×Short 1.14×, long 1.03×Short 2.08×, long 0.18×Short 0.86×, long 0.02×Short 0.30×, long 0.02×
Apply collaborator feedback to a shared road-trip playlist and close the task37.5 s to 1.67 hShort 1.22×, long 1.02×Short 0.94×, long 1.03×Short 1.59×, long 0.13×Short 0.55×, long 0.09×Short 5.92×, long 0.43×Short 1.17×, long 1.03×Short 0.64×, long 0.04×Short 1.02×, long 0.05×Short 1.02×, long 0.02×
Reconcile Venmo payments into Splitwise with attached receipts30 s to 2 hShort 1.09×, long 1.03×Short 1.19×, long 1.04×Short 2.37×, long 0.12×Short 0.33×, long 0.05×Short 5.76×, long 0.83×Short 1.17×, long 1.01×Short 0.75×, long 0.06×Short 0.45×, long 0.03×Short 0.70×, long 0.01×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

The paper’s three agents also ran 4 tasks here that v1.1 replaced: Export every playlist song to a CSV and email the backup, Calculate and send individualized lunch-cost requests with shared tip and delivery, Build scheduled speaker reminders from an email schedule and file template, Send individualized acceptance or rejection replies using local decision templates.

About AppWorld: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by AppWorld's own grader, not a scorer AgentTime invented. The pass rule: Every final-state test passes.

Tasks
14
Asked for
23 s to 5 h
Runs here
276
9 agents
Graded as
Pass or fail: Every final-state test passes.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 7 runs here were stopped by AgentTime, so their time is a lower bound. Details

Data and citation

Upstream: Trivedi et al., 2024 (ACL)