Benchmarks
AgentTime borrows 200 tasks from 18 published benchmarks and adds one sentence to each, asking for a set amount of time.
| ProgramBench | 18 | 1.01× | 2.16× | 2.62× | 1.82× | 2.91× | 1.01× | 10.12× | 13.10× | 45.66× |
| PPTArena | 10 | 1.36× | 2.28× | 3.44× | 2.47× | 3.40× | 1.04× | 3.80× | 5.59× | 31.83× |
| YC-Bench | 4 | 2.22× | 2.06× | 3.43× | 2.10× | 1.98× | 1.05× | 3.23× | 2.89× | 31.49× |
| Agents' Last Exam | 12 | 1.14× | 2.57× | 3.71× | 2.03× | 2.16× | 1.06× | 3.97× | 10.82× | 19.83× |
| METR public tasks | 2 | 1.00× | 4.96× | 5.04× | 1.48× | 3.59× | 1.02× | 4.22× | 9.68× | 11.91× |
| Sakana ALE-Bench | 10 | 1.02× | 1.16× | 1.95× | 1.86× | 1.84× | 1.02× | 3.35× | 4.53× | 23.39× |
| DeepSWE | 14 | 1.08× | 1.76× | 3.49× | 1.79× | 2.77× | 1.17× | 6.26× | 11.44× | 9.05× |
| AppWorld | 14 | 1.09× | 1.15× | 3.63× | 4.02× | 3.95× | 1.09× | 3.61× | 5.97× | 11.50× |
| TUA-Bench | 10 | 1.13× | 1.42× | 3.52× | 2.27× | 3.03× | 1.10× | 4.42× | 6.57× | 11.94× |
| GPQA Diamond | 12 | 1.07× | 1.50× | 4.05× | 1.93× | 3.13× | 1.66× | 4.67× | 14.56× | 2.64× |
| Terminal-Bench 4.0 | 16 | 1.05× | 1.39× | 3.15× | 2.73× | 2.92× | 1.04× | 3.86× | 8.60× | 7.38× |
| CORE-Bench | 12 | 1.04× | 2.02× | 3.33× | 3.52× | 2.87× | 1.04× | 5.65× | 5.08× | 7.46× |
| PostTrainBench | 4 | 1.01× | 2.93× | 6.60× | 1.68× | 2.26× | 1.01× | 2.48× | 6.65× | No runs |
| OSWorld | 18 | 1.20× | 4.48× | 2.30× | 2.10× | 4.04× | 1.62× | 3.24× | 3.08× | 4.25× |
| BrowseComp | 12 | No runs | No runs | No runs | 1.75× | 1.33× | 1.08× | 3.90× | 6.60× | 2.77× |
| Humanity's Last Exam | 20 | 1.09× | 1.28× | 3.04× | 2.61× | 3.19× | 1.61× | 3.91× | 5.56× | 2.44× |
| AssistantBench | 8 | 1.70× | 2.43× | 3.57× | 1.45× | 1.65× | 1.16× | 2.69× | 3.48× | 1.61× |
| PaperBench | 4 | 1.01× | 1.36× | 2.63× | 2.12× | 2.22× | 1.13× | 2.14× | 3.67× | No runs |
Timing error on each benchmark, highest average first. Darker is further from the time asked. Grey values rest on fewer than 10 runs.