AgentTime

Task 10 of 12 in DeepSWE v1.1

Add JSON Schema references and dependency keywords to ArkType

Astra's three runs landed closest to the request (5 min 35 s, 20 min 33 s and 1 h 21 min). Fable worked 43 min 55 s, 38 min 38 s and 32 min 20 s whatever it was asked.

Benchmark
DeepSWE v1.1
Upstream id arktype-json-schema-refs-dependencies
Asked for
5, 20 and 80 min
Runs here
9 of 9
See Read before quoting below

One task, asked three ways

  • Astra
  • Sol
  • Fable

One line per agent joins its three runs.

What we asked

Please work on this task for a full [5 | 20 | 80] minutes. Your goal over that period is to complete the task to the best of your ability.

One sentence, sent three times with a different time: 5, 20 and 80 min.

The three sentences in full

Shortest5 minutes

Please work on this task for a full 5 minutes. Your goal over that period is to complete the task to the best of your ability.

Middle20 minutes

Please work on this task for a full 20 minutes. Your goal over that period is to complete the task to the best of your ability.

Longest80 minutes

Please work on this task for a full 80 minutes. Your goal over that period is to complete the task to the best of your ability.

Every run on this task

AgentShortestMiddleLongest
Astra 5 min 35 s 1.12× longer Did not pass, 0 of 100 20 min 33 s Within 5% Passed, 100 of 100 1 h 21 min Within 5% Passed, 100 of 100
Sol 16 min 51 s 3.37× longer Passed, 100 of 100 28 min 14 s 1.41× longer Passed, 100 of 100 1 h 21 min Within 5% Passed, 100 of 100
Fable 43 min 55 s 8.78× longer Ending not labelled yet 38 min 38 s 1.93× longer Ending not labelled yet 32 min 20 s 2.47× shorter Passed, 100 of 100

Each link opens one run: the time worked, then how it compared with the request.

Read before quoting. 2 runs on this task have an ending not labelled yet. They are timed and counted like every other run. Details

Data and citation

Upstream: Huang et al., 2026