Benchmarks

AgentTime borrows 222 tasks from 18 published benchmarks and adds one sentence to each, asking for a set amount of time.

Timing error for the three paper agents on each of the 18 benchmarks, highest average first

METR public tasks2 h to 60 h	2	1.00× (6 runs)	5.72× (6 runs)	4.11× (6 runs)
PostTrainBench v1.12 h to 30 h	2	1.01× (6 runs)	2.05× (6 runs)	4.98× (5 runs)
WildClawBench1 min to 40 min	10	1.62×	2.72×	3.41×
YC-Bench8 min to 2 h 5 min	3	2.73× (9 runs)	1.75× (9 runs)	2.43× (9 runs)
AssistantBench1 min 15 s to 20 min	18	1.68×	2.16×	2.75×
GPQA Diamond1 min 15 s to 20 min	28	1.06×	1.34×	3.76×
PPTArena1 min 30 s to 25 min	10	1.25×	1.89×	2.89×
OSWorld 2.015 min to 4 h 10 min	16	1.15×	2.88×	1.99×
Agents' Last Exam4 min to 2 h 30 min	12	1.10×	2.20×	2.68×
CORE-Bench v1.12 min 30 s to 3 h 20 min	12	1.04×	1.70×	2.82×
ProgramBench8 min to 8 h 20 min	14	1.01×	1.68×	2.75×
TUA-Bench1 min 15 s to 1 h 40 min	12	1.12×	1.31×	3.01×
DeepSWE v1.14 min to 2 h 30 min	12	1.07×	1.50×	2.57×
AppWorld1 min 30 s to 40 min	18	1.08×	1.13×	2.93×
PaperBench5 min to 3 h 20 min	3	1.01× (9 runs)	1.31× (9 runs)	2.64× (9 runs)
Humanity's Last Exam1 min 15 s to 20 min	24	1.07×	1.21×	2.62×
Terminal-Bench 4.04 min to 8 h 20 min	16	1.04×	1.27×	2.49×
Sakana ALE-Bench10 min to 4 h 10 min	10	1.02×	1.12×	1.97×

Each benchmark counts equally toward an agent's overall number, so PostTrainBench's 2 tasks weigh as much as GPQA Diamond's 28. Grey values rest on fewer than 10 runs.