Benchmark 18 of 18, 2 tasks, asked for 8 min 45 s to 60 h

METR public tasks

Two of METR's public autonomy tasks: building an AI to play a novel board game, and building fraud-style duplicate-payment detection across currencies and time zones.

By agent

Agents on METR public tasks ranked by the share of their runs within 5% of the time asked, with timing error, where the runs ended and score
RankAgentWithin 5%0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming erroron this benchmarkScoreout of 100
1 GPT 6 AstraCodex 100%4 of 4 runs 1.00× Too few graded
2 GPT 6.1 SolCodex 75%3 of 4 runs 1.02× Too few graded
3 GPT 5.6 SolCodex 25%1 of 4 runs 4.96× Too few graded
4 Claude Opus 5.5Claude Code 0%0 of 4 runs 1.48× Too few graded
5 Gemini 4 ArgonAntigravity CLI 0%0 of 4 runs 3.59× Too few graded
6 MiMo V2.6 ProMiMo Code 0%0 of 4 runs 4.22× Too few graded
7 Claude Fable 5.1Claude Code 0%0 of 4 runs 5.04× Too few graded
8 MiMo V2.6 Pro UltraSpeedMiMo Code 0%0 of 4 runs 9.68× Too few graded
9 Muse Wonder 1.0Muse Code 0%0 of 3 runs 11.91× Too few graded

Score is the mean of each agent’s graded runs here, out of 100: a v1.1 agent’s runs with no time asked, the paper’s three at their middle request. How it is measured

Every METR public tasks run

Zoom the chart
View
Typical run, by ask
Short ask
Long ask

Muse Wonder's typical run lasted 1.15× the time asked at the short ask and 0.01× at the long one. Astra's lasted 1.01× and 1.00×.

Hover a mark to read it, click to pin it here.

Showing 35 runs.

The 2 tasks

The 2 tasks on METR public tasks, with where each agent's runs ended
TaskAsked forAstraSolFableOpus 5.5ArgonSol 6.1MiMo ProMiMo UltraSpeedMuse Wonder
Build an efficient AI for the novel Cowthello board game8.75 min to 30 hShort 1.01×, long 1.00×Short 1.07×, long 0.20×Short 0.31×, long 0.28×Short 0.73×, long 0.78×Short 4.34×, long 0.18×Short 1.07×, long 1.00×Short 1.05×, long 0.10×Short 0.51×, long 0.14×Short 2.18×, long no run
Implement duplicate-payment detection across currencies and time zones8.75 min to 60 hShort 1.00×, long 1.00×Short 1.00×, long 0.01×Short 0.34×, long 0.05×Short 0.64×, long 0.58×Short 1.70×, long 0.25×Short 1.01×, long 1.01×Short 0.80×, long 0.04×Short 0.18×, long 0.01×Short 0.12×, long 0.01×

Each mark is one run, at its short or its long ask, placed from 0.1× to 10× of the time asked; the pale band is within 5% of it.

About METR public tasks: the prompt, the grader, the source

Every task's prompt ends with the same sentence for every agent, with that agent's own short or long time:

Please work on this task for a full [time]. Your goal over that period is to complete the task to the best of your ability.

Every run is graded by METR public tasks's own grader, not a scorer AgentTime invented. It reports a score: Cowthello: win rate against 3 bots, divided by a 70% target and capped at 1. Complex Payments: share of held-out feature tests passed.

Tasks
2
Asked for
8 min 45 s to 60 h
Runs here
35
9 agents
Graded as
Cowthello: win rate against 3 bots, divided by a 70% target and capped at 1. Complex Payments: share of held-out feature tests passed.
Data release
2 Nov 2026
17:20 UTC

Read before quoting. 1 run here was stopped by AgentTime, so its time is a lower bound. Details

Data and citation

Upstream: Kinniment et al., 2024 (METR)