Benchmarks

AgentTime borrows 200 tasks from 18 published benchmarks and adds one sentence to each, asking for a set amount of time.

Timing error on each of the 18 benchmarks for the 9 agents, highest average first.
ProgramBench 18 1.01×2.16×2.62×1.82×2.91×1.01×10.12×13.10×45.66×
PPTArena 10 1.36×2.28×3.44×2.47×3.40×1.04×3.80×5.59×31.83×
YC-Bench 4 2.22×2.06×3.43×2.10×1.98×1.05×3.23×2.89×31.49×
Agents' Last Exam 12 1.14×2.57×3.71×2.03×2.16×1.06×3.97×10.82×19.83×
METR public tasks 2 1.00×4.96×5.04×1.48×3.59×1.02×4.22×9.68×11.91×
Sakana ALE-Bench 10 1.02×1.16×1.95×1.86×1.84×1.02×3.35×4.53×23.39×
DeepSWE 14 1.08×1.76×3.49×1.79×2.77×1.17×6.26×11.44×9.05×
AppWorld 14 1.09×1.15×3.63×4.02×3.95×1.09×3.61×5.97×11.50×
TUA-Bench 10 1.13×1.42×3.52×2.27×3.03×1.10×4.42×6.57×11.94×
GPQA Diamond 12 1.07×1.50×4.05×1.93×3.13×1.66×4.67×14.56×2.64×
Terminal-Bench 4.0 16 1.05×1.39×3.15×2.73×2.92×1.04×3.86×8.60×7.38×
CORE-Bench 12 1.04×2.02×3.33×3.52×2.87×1.04×5.65×5.08×7.46×
PostTrainBench 4 1.01×2.93×6.60×1.68×2.26×1.01×2.48×6.65×No runs
OSWorld 18 1.20×4.48×2.30×2.10×4.04×1.62×3.24×3.08×4.25×
BrowseComp 12 No runsNo runsNo runs1.75×1.33×1.08×3.90×6.60×2.77×
Humanity's Last Exam 20 1.09×1.28×3.04×2.61×3.19×1.61×3.91×5.56×2.44×
AssistantBench 8 1.70×2.43×3.57×1.45×1.65×1.16×2.69×3.48×1.61×
PaperBench 4 1.01×1.36×2.63×2.12×2.22×1.13×2.14×3.67×No runs

Timing error on each benchmark, highest average first. Darker is further from the time asked. Grey values rest on fewer than 10 runs.