AgentTime
Measuring Time Awareness in AI agents
200 tasks from 18 benchmarksv1.1, 600 runs per agent and three tests: duration following, forecasting and retrospection.
Combined score over the three tests
- MiMo V2.6 Pro UltraSpeed 6%
- Muse Wonder 1.0 14%
- MiMo V2.6 Pro 18%
- Gemini 4 Argon 19%
- Opus 5.5 20%
- Fable 5.1 26%
- GPT 5.6 Sol 27%
- GPT 6.1 Sol 41%
- GPT 6 Astra 51%
Duration following
Does an AI agent work for as long as it is asked? Each of the 200 tasks was asked for a short and a long duration.
Leaderboard
| Rank | Agent | 0.95× to 1.05× of the request | Where runs ended, from a tenth of the time asked to ten times it | Timing errorlower is better |
|---|---|---|---|---|
| 1 | GPT 6.1 SolCodex, max reasoning | 65%258 runs | 1.15×1.12 to 1.17 | |
| 2 | GPT 6 AstraCodex, max reasoning | 61%242 runs | 1.16×1.10 to 1.21 | |
| 3 | GPT 5.6 SolCodex, max reasoning | 37%149 runs | 1.97×1.76 to 2.16 | |
| 4 | Gemini 4 ArgonAntigravity CLI, high reasoning | 11%42 runs | 2.62×2.44 to 2.81 | |
| 5 | Claude Opus 5.5Claude Code, max reasoning | 6%22 runs | 2.13×2.03 to 2.24 | |
| 6 | Muse Wonder 1.0Muse Code, max reasoning | 5%16 runs | 9.07×7.70 to 10.26 | |
| 7 | Claude Fable 5.1Claude Code, max reasoning | 4%17 runs | 3.37×3.06 to 3.77 | |
| 8 | MiMo V2.6 ProMiMo Code, default reasoning | 4%15 runs | 3.93×3.63 to 4.23 | |
| 9 | MiMo V2.6 Pro UltraSpeedMiMo Code, default reasoning | 3%13 runs | 6.34×5.59 to 7.20 |
The ranges under timing error are 95% intervals. How it is measured
Every run
Lines are the best fits for Sol 6.1, Opus 5.5, Argon, MiMo Pro and Muse Wonder, one per maker. Press an agent to see its runs instead. Hover a mark to read it, click to pin it.
Two speeds
MiMo V2.6 Pro UltraSpeed is MiMo V2.6 Pro served faster. With no time asked it finished the same tasks in under half the time, yet forecast them about the same, so it stopped sooner at every ask.
MiMo ProMiMo UltraSpeed
- No time askedagainst its forecast MiMo Pro 0.30×MiMo UltraSpeed 0.11×
- Asked for a quarterof its forecast MiMo Pro 1.18×MiMo UltraSpeed 0.54×
- Asked for four timesits forecast MiMo Pro 0.13×MiMo UltraSpeed 0.05×
By benchmark
| ProgramBench | 18 | 1.01× | 2.16× | 2.62× | 1.82× | 2.91× | 1.01× | 10.12× | 13.10× | 45.66× |
| PPTArena | 10 | 1.36× | 2.28× | 3.44× | 2.47× | 3.40× | 1.04× | 3.80× | 5.59× | 31.83× |
| YC-Bench | 4 | 2.22× | 2.06× | 3.43× | 2.10× | 1.98× | 1.05× | 3.23× | 2.89× | 31.49× |
| Agents' Last Exam | 12 | 1.14× | 2.57× | 3.71× | 2.03× | 2.16× | 1.06× | 3.97× | 10.82× | 19.83× |
| METR public tasks | 2 | 1.00× | 4.96× | 5.04× | 1.48× | 3.59× | 1.02× | 4.22× | 9.68× | 11.91× |
| Sakana ALE-Bench | 10 | 1.02× | 1.16× | 1.95× | 1.86× | 1.84× | 1.02× | 3.35× | 4.53× | 23.39× |
| DeepSWE | 14 | 1.08× | 1.76× | 3.49× | 1.79× | 2.77× | 1.17× | 6.26× | 11.44× | 9.05× |
| AppWorld | 14 | 1.09× | 1.15× | 3.63× | 4.02× | 3.95× | 1.09× | 3.61× | 5.97× | 11.50× |
| TUA-Bench | 10 | 1.13× | 1.42× | 3.52× | 2.27× | 3.03× | 1.10× | 4.42× | 6.57× | 11.94× |
| GPQA Diamond | 12 | 1.07× | 1.50× | 4.05× | 1.93× | 3.13× | 1.66× | 4.67× | 14.56× | 2.64× |
| Terminal-Bench 4.0 | 16 | 1.05× | 1.39× | 3.15× | 2.73× | 2.92× | 1.04× | 3.86× | 8.60× | 7.38× |
| CORE-Bench | 12 | 1.04× | 2.02× | 3.33× | 3.52× | 2.87× | 1.04× | 5.65× | 5.08× | 7.46× |
| PostTrainBench | 4 | 1.01× | 2.93× | 6.60× | 1.68× | 2.26× | 1.01× | 2.48× | 6.65× | No runs |
| OSWorld | 18 | 1.20× | 4.48× | 2.30× | 2.10× | 4.04× | 1.62× | 3.24× | 3.08× | 4.25× |
| BrowseComp | 12 | No runs | No runs | No runs | 1.75× | 1.33× | 1.08× | 3.90× | 6.60× | 2.77× |
| Humanity's Last Exam | 20 | 1.09× | 1.28× | 3.04× | 2.61× | 3.19× | 1.61× | 3.91× | 5.56× | 2.44× |
| AssistantBench | 8 | 1.70× | 2.43× | 3.57× | 1.45× | 1.65× | 1.16× | 2.69× | 3.48× | 1.61× |
| PaperBench | 4 | 1.01× | 1.36× | 2.63× | 2.12× | 2.22× | 1.13× | 2.14× | 3.67× | No runs |
Timing error on each benchmark, highest average first. Darker is further from the time asked. Grey values rest on fewer than 10 runs. All 18 benchmarks
Forecasting
Can an agent say how long a task will take it? Each agent forecast all 200 tasks before starting, then ran each one with no time asked.
| Agent | Most forecasts | Typical miss | Median forecast | Median runtime | Benchmark score |
|---|---|---|---|---|---|
| Claude Opus 5.5 | 58% too low | 2.26×, the smallest | 30 min | 32 min | 72.2, the highest67.2 to 77.1 |
| Claude Fable 5.1 | 63% too high | 2.47× | 15 min | 14 min | 60.054.6 to 64.6 |
| Gemini 4 Argon | 79% too low | 2.59× | 12 min | 25 min | 68.462.9 to 74.0 |
| GPT 6.1 Sol | 90% too high | 2.85× | 25 min | 9 min | 67.863.3 to 72.5 |
| GPT 6 Astra | 63% too high | 2.96× | 20 min | 11 min | 64.659.2 to 69.5 |
| GPT 5.6 Sol | 84% too high | 3.82× | 15 min | 7 min | 56.451.1 to 61.5 |
| MiMo V2.6 Pro | 79% too high | 4.19× | 60 min | 30 min | 48.042.9 to 53.1 |
| Muse Wonder 1.0 | 92% too high | 8.44× | 40 min | 8 min | 57.150.4 to 63.5 |
| MiMo V2.6 Pro UltraSpeed | 90% too high | 9.42× | 71 min | 11 min | 50.745.4 to 55.8 |
Typical miss is the geometric mean of the factor between a forecast and the agent's own runtime on that task. Darker is further off. Claude Fable 5.1 gave no number for 6 tasks. Benchmark score is out of 100, from the same runs (for the paper's three agents, their middle request, about a task's usual time), each benchmark counting equally; PaperBench and PostTrainBench are not in it. Small figures are 95% intervals. Paper, Section 4.2
Told their speed
Told its own output speed in one added line, every agent forecast shorter, by 17 to 31%. That helped the two that forecast too long and hurt the two that already forecast too short.
Asked as usualTold its speed
- Muse Wonder 1.0miss 8.08× → 6.57× as usual, 7.35× its runtimetold its speed, 5.40× its runtime
- GPT 6.1 Solmiss 2.83× → 2.14× as usual, 2.64× its runtimetold its speed, 1.83× its runtime
- Claude Opus 5.5miss 2.21× → 2.32× as usual, 0.85× its runtimetold its speed, 0.67× its runtime
- Gemini 4 Argonmiss 3.07× → 3.37× as usual, 0.49× its runtimetold its speed, 0.40× its runtime
Retrospection
Can an agent tell how long it has worked? Each agent estimated how long its own finished runs took, with less time information at each step.
| Agent | Oracle | Native | Context-only | Replay | Scrubbed |
|---|---|---|---|---|---|
| GPT 6 Astra | 1.00× | 1.10× | 1.22× | 1.20× | 1.39× |
| Claude Fable 5.1 | 1.00× | 1.32× | 1.63× | 1.55× | 1.51× |
| GPT 6.1 Sol | 1.00× | 1.46× | 1.51× | 1.86× | 1.91× |
| MiMo V2.6 Pro | 1.00× | 1.50× | 1.67× | 1.75× | 2.04× |
| GPT 5.6 Sol | 1.00× | 1.64× | 1.53× | 1.87× | 2.02× |
| Muse Wonder 1.0 | 1.01× | 1.65× | 1.87× | 2.96× | 3.58× |
| Claude Opus 5.5 | 1.01× | 1.71× | 2.12× | 1.34× | 1.77× |
| Gemini 4 Argon | 1.00× | 1.75× | 2.47× | 2.06× | 3.73× |
| MiMo V2.6 Pro UltraSpeed | 1.01× | 2.81× | 3.06× | 3.52× | 3.87× |
Typical miss against the real runtime on 23 tasks, two answers per condition. Oracle adds an elapsed-time tool. Native and Context-only continue the finished session, with tools on or off. Replay sends a rebuilt transcript to the model's API, and Scrubbed does the same without timestamps or other time cues. Darker is further off. MiMo V2.6 Pro was stopped early, so its row covers 21 tasks. Muse Wonder 1.0's row covers 21 tasks. Paper, Section 4.3