Benchmark 9 of 18, 8 tasks, asked for 53 s to 2 h
AssistantBench
Open-ended questions that can only be answered by browsing the live web, such as tracking down a specific number or fact across several pages.
By agent
| Rank | Agent | Within 5%0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing erroron this benchmark | Scoreout of 100 |
|---|---|---|---|---|---|
| 1 | GPT 6.1 SolCodex | 50%8 of 16 runs | 1.16× | 6.28 graded | |
| 2 | Gemini 4 ArgonAntigravity CLI | 38%6 of 16 runs | 1.65× | 40.78 graded | |
| 3 | Muse Wonder 1.0Muse Code | 31%5 of 16 runs | 1.61× | 17.38 graded | |
| 4 | Claude Opus 5.5Claude Code | 25%4 of 16 runs | 1.45× | 16.08 graded | |
| 5 | GPT 6 AstraCodex | 19%3 of 16 runs | 1.70× | 3.68 graded | |
| 6 | GPT 5.6 SolCodex | 13%2 of 16 runs | 2.43× | 4.58 graded | |
| 7 | MiMo V2.6 Pro UltraSpeedMiMo Code | 13%2 of 16 runs | 3.48× | 13.38 graded | |
| 8 | MiMo V2.6 ProMiMo Code | 6%1 of 16 runs | 2.69× | 7.58 graded | |
| 9 | Claude Fable 5.1Claude Code | 0%0 of 16 runs | 3.57× | 2.68 graded |
Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured
Every AssistantBench run
Fable's typical run lasted 2.11× the time asked at the short ask and 0.35× at the long one. Sol 6.1's lasted 1.18× and 1.02×.
Hover a mark to read it, click to pin it here.
Showing 144 runs.
The 8 tasks
| Task | Asked for | Astra | Sol | Fable | Opus 5.5 | Argon | Sol 6.1 | MiMo Pro | MiMo UltraSpeed | Muse Wonder |
|---|---|---|---|---|---|---|---|---|---|---|
| Task 3 | 52.5 s to 2 h | Short 1.34×, long 0.29× | Short 4.96×, long 0.93× | Short 2.50×, long 0.40× | Short 1.20×, long 1.01× | Short 1.58×, long 1.06× | Short 1.10×, long 1.02× | Short 3.14×, long 0.22× | Short 1.03×, long 0.11× | Short 1.14×, long 0.36× |
| Task 6 | 1.25 min to 2 h | Short 1.92×, long 0.94× | Short 6.31×, long 0.63× | Short 32.00×, long 2.00× | Short 2.01×, long 0.55× | Short 1.26×, long 1.44× | Short 1.13×, long 1.02× | Short 4.23×, long 1.02× | Short 7.95×, long 0.64× | Short 1.52×, long 1.08× |
| Task 7 | 1.13 to 56 min | Short 1.33×, long 1.02× | Short 4.66×, long 0.94× | Short 8.88×, long 0.59× | Short 1.30×, long 1.02× | Short 7.07×, long 1.05× | Short 1.16×, long 1.03× | Short 2.81×, long 0.65× | Short 2.03×, long 0.27× | Short 3.92×, long 1.05× |
| Task 11 | 1.13 min to 2 h | Short 1.42×, long 1.04× | Short 1.51×, long 0.94× | Short 1.72×, long 0.13× | Short 1.14×, long 1.01× | Short 12.16×, long 1.03× | Short 1.08×, long 1.01× | Short 2.93×, long 0.26× | Short 0.53×, long 0.07× | Short 1.03×, long 0.28× |
| Task 14 | 1.25 min to 1.47 h | Short 1.52×, long 0.35× | Short 12.07×, long 0.68× | Short 1.06×, long 0.23× | Short 1.40×, long 0.87× | Short 1.46×, long 1.03× | Short 1.20×, long 1.01× | Short 1.16×, long 0.41× | Short 1.05×, long 0.23× | Short 1.80×, long 0.47× |
| Task 16 | 52.5 s to 2 h | Short 2.38×, long 0.24× | Short 6.76×, long 0.97× | Short 1.66×, long 0.22× | Short 1.18×, long 0.46× | Short 1.19×, long 1.05× | Short 1.45×, long 1.03× | Short 2.88×, long 0.52× | Short 0.96×, long 0.11× | Short 1.59×, long 1.02× |
| Task 18 | 1.25 min to 2 h | Short 3.44×, long 0.69× | Short 7.77×, long 0.71× | Short 32.00×, long 2.00× | Short 3.60×, long 0.40× | Short 2.85×, long 1.02× | Short 2.65×, long 1.02× | Short 5.19×, long 0.55× | Short 3.53×, long 3.00× | Short 1.91×, long 1.01× |
| Task 17 | 52.5 s to 1.67 h | Short 1.15×, long 1.03× | Short 2.39×, long 1.03× | Short 1.19×, long 0.30× | Short 1.48×, long 1.03× | Short 1.92×, long 1.05× | Short 1.32×, long 1.03× | Short 4.12×, long 0.24× | Short 0.16×, long 0.14× | Short 2.13×, long 1.01× |
No runs match these filters.
Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.
About AssistantBench: the prompt, the grader, the source
Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:
Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.
Every run is graded by AssistantBench's own grader, not a scorer AgentTime invented. The pass rule: Exact match with the reference answer.
- Tasks
- 8
- Asked for
- 53 s to 2 h
- Runs here
- 144
- 9 agents
- Graded as
- Pass or fail: Exact match with the reference answer.
- Data release
- 2 Nov 2026
- 17:20 UTC
Data and citation
Upstream: Yoran et al., 2024 (EMNLP)