AgentTime

Task 6 of 14 in ProgramBench

ninja-build/ninja

Astra's three runs followed the request (10 min 18 s, 40 min 21 s and 3 h 21 min). Fable's runs landed furthest from the request (1 h 53 min, 26 min 59 s and 1 h 38 min).

Benchmark
ProgramBench
Upstream id ninja-build__ninja.cc60300
Asked for
10, 40 and 200 min
Runs here
9 of 9
See Read before quoting below

One task, asked three ways

  • Astra
  • Sol
  • Fable

One line per agent joins its three runs.

What we asked

Please work on this task for a full [10 | 40 | 200] minutes. Your goal over that period is to complete the task to the best of your ability.

One sentence, sent three times with a different time: 10, 40 and 200 min.

The three sentences in full

Shortest10 minutes

Please work on this task for a full 10 minutes. Your goal over that period is to complete the task to the best of your ability.

Middle40 minutes

Please work on this task for a full 40 minutes. Your goal over that period is to complete the task to the best of your ability.

Longest200 minutes

Please work on this task for a full 200 minutes. Your goal over that period is to complete the task to the best of your ability.

Every run on this task

AgentShortestMiddleLongest
Astra 10 min 18 s Within 5% Ending not labelled yet 40 min 21 s Within 5% Ending not labelled yet 3 h 21 min Within 5% Ending not labelled yet
Sol 23 min 54 s 2.39× longer Ending not labelled yet 40 min 17 s Within 5% Ending not labelled yet 37 min 32 s 5.33× shorter Ending not labelled yet
Fable 1 h 53 min 11.30× longer Did not pass, 93.1 of 100 26 min 59 s 1.48× shorter Did not pass, 85.3 of 100 1 h 38 min 2.05× shorter Did not pass, 95.8 of 100

Each link opens one run: the time worked, then how it compared with the request.

Read before quoting. 6 runs on this task have an ending not labelled yet. They are timed and counted like every other run. Details

Data and citation

Upstream: Yang et al., 2026