Benchmark 5 of 18, 14 tasks, asked for 23 s to 5 h
AppWorld
Simulated everyday app tasks, like exporting a playlist to a spreadsheet or reconciling an emailed song list with what is actually on Spotify, done by calling app APIs.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6 AstraCodex | 50%18 of 36 runs | 1.09× | 66.718 graded | |
| 2 | GPT 6.1 SolCodex | 50%14 of 28 runs | 1.09× | 57.114 graded | |
| 3 | GPT 5.6 SolCodex | 36%13 of 36 runs | 1.15× | 66.718 graded | |
| 4 | MiMo V2.6 ProMiMo Code | 11%3 of 28 runs | 3.61× | 35.714 graded | |
| 5 | Gemini 4 ArgonAntigravity CLI | 7%2 of 28 runs | 3.95× | 57.114 graded | |
| 6 | Muse Wonder 1.0Muse Code | 7%2 of 28 runs | 11.50× | 50.014 graded | |
| 7 | MiMo V2.6 Pro UltraSpeedMiMo Code | 4%1 of 28 runs | 5.97× | 35.714 graded | |
| 8 | Claude Fable 5.1Claude Code | 0%0 of 36 runs | 3.63× | 72.218 graded | |
| 9 | Claude Opus 5.5Claude Code | 0%0 of 28 runs | 4.02× | 57.114 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every AppWorld run
Muse Wonder's typical run lasted 0.79× the time asked at the short ask and 0.02× at the long one. Astra's lasted 1.15× and 1.03×.
Hover a mark to read it, click to pin it here.
Showing 276 runs.
The 14 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| Execute every task in a camping project across shopping, expenses, payments, invitations, and task closeout | 1.5 min to 5 h | Short 1.08×, long 1.02× | Short 1.33×, long 1.02× | Short 2.74×, long 0.19× | Short 1.88×, long 0.22× | Short 32.00×, long 2.00× | Short 1.09×, long 1.01× | Short 2.37×, long 1.00× | Short 1.55×, long 0.12× | Short 0.15×, long 0.04× |
| Reconcile an emailed road-trip song list with a Spotify playlist and download it | 22.5 s to 1.2 h | Short 1.12×, long 1.03× | Short 2.12×, long 1.05× | Short 1.92×, long 0.14× | Short 0.45×, long 0.05× | Short 12.29×, long 0.89× | Short 1.21×, long 1.02× | Short 0.99×, long 1.10× | Short 2.03×, long 0.05× | Short 1.30×, long 0.04× |
| Turn this year’s emailed electricity bills into monthly Splitwise expenses | 37.5 s to 3 h | Short 1.16×, long 1.02× | Short 1.59×, long 1.03× | Short 1.96×, long 0.14× | Short 0.58×, long 0.09× | Short 7.10×, long 0.96× | Short 1.17×, long 1.02× | Short 0.62×, long 0.04× | Short 0.47×, long 0.03× | Short 1.03×, long 0.03× |
| Buy a requested item, forward its receipt, and recover the cost through Venmo | 37.5 s to 1.87 h | Short 1.24×, long 1.04× | Short 1.13×, long 1.05× | Short 2.22×, long 0.16× | Short 2.06×, long 0.22× | Short 32.00×, long 2.00× | Short 1.15×, long 1.03× | Short 4.40×, long 0.53× | Short 1.18×, long 0.17× | Short 0.04×, long 0.00× |
| Collect artist recommendations from email and text conversations, then follow them | 30 s to 1.87 h | Short 1.11×, long 1.02× | Short 1.15×, long 1.03× | Short 1.46×, long 0.12× | Short 0.72×, long 0.05× | Short 9.84×, long 0.89× | Short 1.13×, long 1.02× | Short 0.75×, long 0.04× | Short 1.18×, long 0.06× | Short 0.41×, long 0.01× |
| Rebuild and send a drafted recommendation email from the user’s Spotify history | 37.5 s to 1.67 h | Short 1.20×, long 1.04× | Short 1.24×, long 1.05× | Short 1.30×, long 0.15× | Short 0.62×, long 0.06× | Short 5.94×, long 0.56× | Short 1.12×, long 1.02× | Short 1.05×, long 0.07× | Short 0.26×, long 0.02× | Short 0.88×, long 0.03× |
| Merge two photo collections into a ZIP and send it to both parents | 30 s to 1.67 h | Short 1.09×, long 1.02× | Short 1.17×, long 1.03× | Short 1.75×, long 0.14× | Short 0.83×, long 0.09× | Short 7.14×, long 1.17× | Short 1.21×, long 1.03× | Short 1.26×, long 0.09× | Short 0.23×, long 0.02× | Short 1.28×, long 0.04× |
| Reconcile a camping checklist with commitments and buy the remaining supplies | 37.5 s to 3 h | Short 1.14×, long 1.04× | Short 1.15×, long 1.03× | Short 1.94×, long 0.17× | Short 2.11×, long 0.33× | Short 32.00×, long 2.00× | Short 1.14×, long 1.03× | Short 2.88×, long 1.00× | Short 1.53×, long 0.15× | Short 1.12×, long 0.06× |
| Collect every cable bill, normalize filenames, ZIP them, and reply to the request | 37.5 s to 2.8 h | Short 1.08×, long 1.03× | Short 1.19×, long 1.02× | Short 2.04×, long 0.12× | Short 1.07×, long 0.09× | Short 6.10×, long 1.43× | Short 1.14×, long 1.03× | Short 0.44×, long 0.13× | Short 1.09×, long 0.09× | Short 0.05×, long 0.01× |
| Turn a note-based meeting schedule into correctly timed reminder emails | 30 s to 2.33 h | Short 1.16×, long 1.03× | Short 0.84×, long 1.07× | Short 2.11×, long 0.14× | Short 0.81×, long 0.12× | Short 3.62×, long 1.06× | Short 1.24×, long 1.02× | Short 0.70×, long 0.03× | Short 0.38×, long 0.03× | Short 1.05×, long 0.02× |
| Build a recruiting CSV and archive each candidate’s attachments | 37.5 s to 2 h | Short 1.22×, long 1.03× | Short 1.44×, long 1.04× | Short 1.65×, long 0.39× | Short 0.67×, long 0.08× | Short 22.24×, long 2.00× | Short 1.17×, long 1.02× | Short 0.58×, long 0.07× | Short 0.79×, long 0.09× | Short 0.21×, long 0.02× |
| Repack tonight’s Todoist plan around the available work schedule | 45 s to 2.13 h | Short 1.21×, long 1.04× | Short 1.48×, long 1.04× | Short 1.91×, long 0.17× | Short 0.81×, long 0.22× | Short 13.11×, long 1.05× | Short 1.14×, long 1.03× | Short 2.08×, long 0.18× | Short 0.86×, long 0.02× | Short 0.30×, long 0.02× |
| Apply collaborator feedback to a shared road-trip playlist and close the task | 37.5 s to 1.67 h | Short 1.22×, long 1.02× | Short 0.94×, long 1.03× | Short 1.59×, long 0.13× | Short 0.55×, long 0.09× | Short 5.92×, long 0.43× | Short 1.17×, long 1.03× | Short 0.64×, long 0.04× | Short 1.02×, long 0.05× | Short 1.02×, long 0.02× |
| Reconcile Venmo payments into Splitwise with attached receipts | 30 s to 2 h | Short 1.09×, long 1.03× | Short 1.19×, long 1.04× | Short 2.37×, long 0.12× | Short 0.33×, long 0.05× | Short 5.76×, long 0.83× | Short 1.17×, long 1.01× | Short 0.75×, long 0.06× | Short 0.45×, long 0.03× | Short 0.70×, long 0.01× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
The paper’s three agents also ran 4 tasks here that v1.1 replaced: Export every playlist song to a CSV and email the backup, Calculate and send individualized lunch-cost requests with shared tip and delivery, Build scheduled speaker reminders from an email schedule and file template, Send individualized acceptance or rejection replies using local decision templates.
About AppWorld: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by AppWorld's own grader, not a scorer AgentTime invented. The pass rule: Every final-state test passes.
- Tasks
- 14
- Asked for
- 23 s to 5 h
- Runs here
- 276
- 9 agents
- Graded as
- Pass or fail: Every final-state test passes.
- Data release
- 2 Nov 2026
- 17:20 UTC
Read before quoting. 7 runs here were stopped by AgentTime, so their time is a lower bound. Details
Data and citation
Upstream: Trivedi et al., 2024 (ACL)