AgentTime

Measuring Time Awareness in AI agents

200 tasks from 18 benchmarksv1.1, 600 runs per agent and three tests: duration following, forecasting and retrospection.

Combined score over the three tests

Higher is better. How it is counted

Duration following

Does an AI agent work for as long as it is asked? Each of the 200 tasks was asked for a short and a long duration.

Leaderboard

Agents ranked by the share of their runs within 5% of the time asked, higher is better.
RankAgent0.95× to 1.05× of the requestWhere runs ended, from a tenth of the time asked to ten times itTiming errorlower is better
1 GPT 6.1 SolCodex, max reasoning 65%258 runs 1.15×1.12 to 1.17
2 GPT 6 AstraCodex, max reasoning 61%242 runs 1.16×1.10 to 1.21
3 GPT 5.6 SolCodex, max reasoning 37%149 runs 1.97×1.76 to 2.16
4 Gemini 4 ArgonAntigravity CLI, high reasoning 11%42 runs 2.62×2.44 to 2.81
5 Claude Opus 5.5Claude Code, max reasoning 6%22 runs 2.13×2.03 to 2.24
6 Muse Wonder 1.0Muse Code, max reasoning 5%16 runs 9.07×7.70 to 10.26
7 Claude Fable 5.1Claude Code, max reasoning 4%17 runs 3.37×3.06 to 3.77
8 MiMo V2.6 ProMiMo Code, default reasoning 4%15 runs 3.93×3.63 to 4.23
9 MiMo V2.6 Pro UltraSpeedMiMo Code, default reasoning 3%13 runs 6.34×5.59 to 7.20

The ranges under timing error are 95% intervals. How it is measured

Every run

Lines are the best fits for Sol 6.1, Opus 5.5, Argon, MiMo Pro and Muse Wonder, one per maker. Press an agent to see its runs instead. Hover a mark to read it, click to pin it.

Open the results explorer

Two speeds

MiMo V2.6 Pro UltraSpeed is MiMo V2.6 Pro served faster. With no time asked it finished the same tasks in under half the time, yet forecast them about the same, so it stopped sooner at every ask.

MiMo ProMiMo UltraSpeed

  1. No time askedagainst its forecast MiMo Pro 0.30×MiMo UltraSpeed 0.11×
  2. Asked for a quarterof its forecast MiMo Pro 1.18×MiMo UltraSpeed 0.54×
  3. Asked for four timesits forecast MiMo Pro 0.13×MiMo UltraSpeed 0.05×
Median time worked against the time asked, or with no time asked against the agent's own forecast. On the 200 tasks both finished with no time asked, MiMo UltraSpeed was quicker on 179.

By benchmark

Timing error on each of the 18 benchmarks for the 9 agents, highest average first.
ProgramBench 18 1.01×2.16×2.62×1.82×2.91×1.01×10.12×13.10×45.66×
PPTArena 10 1.36×2.28×3.44×2.47×3.40×1.04×3.80×5.59×31.83×
YC-Bench 4 2.22×2.06×3.43×2.10×1.98×1.05×3.23×2.89×31.49×
Agents' Last Exam 12 1.14×2.57×3.71×2.03×2.16×1.06×3.97×10.82×19.83×
METR public tasks 2 1.00×4.96×5.04×1.48×3.59×1.02×4.22×9.68×11.91×
Sakana ALE-Bench 10 1.02×1.16×1.95×1.86×1.84×1.02×3.35×4.53×23.39×
DeepSWE 14 1.08×1.76×3.49×1.79×2.77×1.17×6.26×11.44×9.05×
AppWorld 14 1.09×1.15×3.63×4.02×3.95×1.09×3.61×5.97×11.50×
TUA-Bench 10 1.13×1.42×3.52×2.27×3.03×1.10×4.42×6.57×11.94×
GPQA Diamond 12 1.07×1.50×4.05×1.93×3.13×1.66×4.67×14.56×2.64×
Terminal-Bench 4.0 16 1.05×1.39×3.15×2.73×2.92×1.04×3.86×8.60×7.38×
CORE-Bench 12 1.04×2.02×3.33×3.52×2.87×1.04×5.65×5.08×7.46×
PostTrainBench 4 1.01×2.93×6.60×1.68×2.26×1.01×2.48×6.65×No runs
OSWorld 18 1.20×4.48×2.30×2.10×4.04×1.62×3.24×3.08×4.25×
BrowseComp 12 No runsNo runsNo runs1.75×1.33×1.08×3.90×6.60×2.77×
Humanity's Last Exam 20 1.09×1.28×3.04×2.61×3.19×1.61×3.91×5.56×2.44×
AssistantBench 8 1.70×2.43×3.57×1.45×1.65×1.16×2.69×3.48×1.61×
PaperBench 4 1.01×1.36×2.63×2.12×2.22×1.13×2.14×3.67×No runs

Timing error on each benchmark, highest average first. Darker is further from the time asked. Grey values rest on fewer than 10 runs. All 18 benchmarks

Forecasting

Can an agent say how long a task will take it? Each agent forecast all 200 tasks before starting, then ran each one with no time asked.

Forecasts against each agent's own runtime on the same tasks.
AgentMost forecastsTypical missMedian forecastMedian runtimeBenchmark score
Claude Opus 5.558% too low2.26×, the smallest30 min32 min72.2, the highest67.2 to 77.1
Claude Fable 5.163% too high2.47×15 min14 min60.054.6 to 64.6
Gemini 4 Argon79% too low2.59×12 min25 min68.462.9 to 74.0
GPT 6.1 Sol90% too high2.85×25 min9 min67.863.3 to 72.5
GPT 6 Astra63% too high2.96×20 min11 min64.659.2 to 69.5
GPT 5.6 Sol84% too high3.82×15 min7 min56.451.1 to 61.5
MiMo V2.6 Pro79% too high4.19×60 min30 min48.042.9 to 53.1
Muse Wonder 1.092% too high8.44×40 min8 min57.150.4 to 63.5
MiMo V2.6 Pro UltraSpeed90% too high9.42×71 min11 min50.745.4 to 55.8

Typical miss is the geometric mean of the factor between a forecast and the agent's own runtime on that task. Darker is further off. Claude Fable 5.1 gave no number for 6 tasks. Benchmark score is out of 100, from the same runs (for the paper's three agents, their middle request, about a task's usual time), each benchmark counting equally; PaperBench and PostTrainBench are not in it. Small figures are 95% intervals. Paper, Section 4.2

Told their speed

Told its own output speed in one added line, every agent forecast shorter, by 17 to 31%. That helped the two that forecast too long and hurt the two that already forecast too short.

Asked as usualTold its speed

  1. Muse Wonder 1.0miss 8.08× → 6.57× as usual, 7.35× its runtimetold its speed, 5.40× its runtime
  2. GPT 6.1 Solmiss 2.83× → 2.14× as usual, 2.64× its runtimetold its speed, 1.83× its runtime
  3. Claude Opus 5.5miss 2.21× → 2.32× as usual, 0.85× its runtimetold its speed, 0.67× its runtime
  4. Gemini 4 Argonmiss 3.07× → 3.37× as usual, 0.49× its runtimetold its speed, 0.40× its runtime
Each agent forecast every task again on 3 Nov 2026, twice side by side: as usual, and with one line giving its output speed in tokens per second (172 for Muse Wonder 1.0, 79 for GPT 6.1 Sol, 96 for Claude Opus 5.5 and 82 for Gemini 4 Argon, from Artificial Analysis). Position is the geometric mean of forecast over the agent's own runtime with no time asked; miss is the typical miss. These new asks differ from the table above, and Gemini 4 Argon answered in a text-only mode, so only its text tasks count: 174 to 200 tasks per agent.

Retrospection

Can an agent tell how long it has worked? Each agent estimated how long its own finished runs took, with less time information at each step.

How far each agent's estimate of its own finished runs is from the real runtime, under five conditions.
AgentOracleNativeContext-onlyReplayScrubbed
GPT 6 Astra1.00×1.10×1.22×1.20×1.39×
Claude Fable 5.11.00×1.32×1.63×1.55×1.51×
GPT 6.1 Sol1.00×1.46×1.51×1.86×1.91×
MiMo V2.6 Pro1.00×1.50×1.67×1.75×2.04×
GPT 5.6 Sol1.00×1.64×1.53×1.87×2.02×
Muse Wonder 1.01.01×1.65×1.87×2.96×3.58×
Claude Opus 5.51.01×1.71×2.12×1.34×1.77×
Gemini 4 Argon1.00×1.75×2.47×2.06×3.73×
MiMo V2.6 Pro UltraSpeed1.01×2.81×3.06×3.52×3.87×

Typical miss against the real runtime on 23 tasks, two answers per condition. Oracle adds an elapsed-time tool. Native and Context-only continue the finished session, with tools on or off. Replay sends a rebuilt transcript to the model's API, and Scrubbed does the same without timestamps or other time cues. Darker is further off. MiMo V2.6 Pro was stopped early, so its row covers 21 tasks. Muse Wonder 1.0's row covers 21 tasks. Paper, Section 4.3