Benchmark 9 of 18, 8 tasks, asked for 53 s to 2 h

AssistantBench

Open-ended questions that can only be answered by browsing the live web, such as tracking down a specific number or fact across several pages.

By agent

Agents on AssistantBench ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6.1 SolCodex 50%8 of 16 runs 1.16× 6.28 graded
2 Gemini 4 ArgonAntigravity CLI 38%6 of 16 runs 1.65× 40.78 graded
3 Muse Wonder 1.0Muse Code 31%5 of 16 runs 1.61× 17.38 graded
4 Claude Opus 5.5Claude Code 25%4 of 16 runs 1.45× 16.08 graded
5 GPT 6 AstraCodex 19%3 of 16 runs 1.70× 3.68 graded
6 GPT 5.6 SolCodex 13%2 of 16 runs 2.43× 4.58 graded
7 MiMo V2.6 Pro UltraSpeedMiMo Code 13%2 of 16 runs 3.48× 13.38 graded
8 MiMo V2.6 ProMiMo Code 6%1 of 16 runs 2.69× 7.58 graded
9 Claude Fable 5.1Claude Code 0%0 of 16 runs 3.57× 2.68 graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every AssistantBench run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Fable's typical run lasted 2.11× the time asked at the short ask and 0.35× at the long one. Sol 6.1's lasted 1.18× and 1.02×.

Hover a mark to read it, click to pin it here.

Showing 144 runs.

The 8 tasks

The 8 tasks on AssistantBench, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
Task 352.5 s to 2 hShort 1.34×, long 0.29×Short 4.96×, long 0.93×Short 2.50×, long 0.40×Short 1.20×, long 1.01×Short 1.58×, long 1.06×Short 1.10×, long 1.02×Short 3.14×, long 0.22×Short 1.03×, long 0.11×Short 1.14×, long 0.36×
Task 61.25 min to 2 hShort 1.92×, long 0.94×Short 6.31×, long 0.63×Short 32.00×, long 2.00×Short 2.01×, long 0.55×Short 1.26×, long 1.44×Short 1.13×, long 1.02×Short 4.23×, long 1.02×Short 7.95×, long 0.64×Short 1.52×, long 1.08×
Task 71.13 to 56 minShort 1.33×, long 1.02×Short 4.66×, long 0.94×Short 8.88×, long 0.59×Short 1.30×, long 1.02×Short 7.07×, long 1.05×Short 1.16×, long 1.03×Short 2.81×, long 0.65×Short 2.03×, long 0.27×Short 3.92×, long 1.05×
Task 111.13 min to 2 hShort 1.42×, long 1.04×Short 1.51×, long 0.94×Short 1.72×, long 0.13×Short 1.14×, long 1.01×Short 12.16×, long 1.03×Short 1.08×, long 1.01×Short 2.93×, long 0.26×Short 0.53×, long 0.07×Short 1.03×, long 0.28×
Task 141.25 min to 1.47 hShort 1.52×, long 0.35×Short 12.07×, long 0.68×Short 1.06×, long 0.23×Short 1.40×, long 0.87×Short 1.46×, long 1.03×Short 1.20×, long 1.01×Short 1.16×, long 0.41×Short 1.05×, long 0.23×Short 1.80×, long 0.47×
Task 1652.5 s to 2 hShort 2.38×, long 0.24×Short 6.76×, long 0.97×Short 1.66×, long 0.22×Short 1.18×, long 0.46×Short 1.19×, long 1.05×Short 1.45×, long 1.03×Short 2.88×, long 0.52×Short 0.96×, long 0.11×Short 1.59×, long 1.02×
Task 181.25 min to 2 hShort 3.44×, long 0.69×Short 7.77×, long 0.71×Short 32.00×, long 2.00×Short 3.60×, long 0.40×Short 2.85×, long 1.02×Short 2.65×, long 1.02×Short 5.19×, long 0.55×Short 3.53×, long 3.00×Short 1.91×, long 1.01×
Task 1752.5 s to 1.67 hShort 1.15×, long 1.03×Short 2.39×, long 1.03×Short 1.19×, long 0.30×Short 1.48×, long 1.03×Short 1.92×, long 1.05×Short 1.32×, long 1.03×Short 4.12×, long 0.24×Short 0.16×, long 0.14×Short 2.13×, long 1.01×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

About AssistantBench: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by AssistantBench's own grader, not a scorer AgentTime invented. The pass rule: Exact match with the reference answer.

Tasks
8
Asked for
53 s to 2 h
Runs here
144
9 agents
Graded as
Pass or fail: Exact match with the reference answer.
Data release
2 Nov 2026
17:20 UTC

Data and citation

Upstream: Yoran et al., 2024 (EMNLP)