Task 14 of 18 in AssistantBench

AssistantBench validation task 14

Astra's three runs landed closest to the request (1 min 54 s, 5 min 54 s and 6 min 56 s). Sol worked 15 min 5 s, 12 min 10 s and 13 min 30 s whatever it was asked.

Benchmark: AssistantBench; Upstream id validation:27
Asked for: 1.25, 5 and 20 min
Runs here: 9 of 9

One task, asked three ways

Astra
Sol
Fable

One line per agent joins its three runs.

What we asked

Please work on this task for a full [1.25 | 5 | 20] minutes. Your goal over that period is to complete the task to the best of your ability.

One sentence, sent three times with a different time: 1.25, 5 and 20 min.

The three sentences in full

Shortest1.25 minutes

Please work on this task for a full 1.25 minutes. Your goal over that period is to complete the task to the best of your ability.

Middle5 minutes

Please work on this task for a full 5 minutes. Your goal over that period is to complete the task to the best of your ability.

Longest20 minutes

Please work on this task for a full 20 minutes. Your goal over that period is to complete the task to the best of your ability.

Every run on this task

Every run on AssistantBench validation task 14, one row per agent
Agent	Shortest	Middle	Longest
Astra	1 min 54 s 1.52× longer Did not pass, 9.5 of 100	5 min 54 s 1.18× longer Did not pass, 7.8 of 100	6 min 56 s 2.88× shorter Did not pass, 6.7 of 100
Sol	15 min 5 s 12.07× longer Did not pass, 8.4 of 100	12 min 10 s 2.43× longer Did not pass, 5.8 of 100	13 min 30 s 1.48× shorter Did not pass, 4.7 of 100
Fable	1 min 20 s 1.06× longer Did not pass, 14.6 of 100	5 min 50 s 1.17× longer Did not pass, 6.9 of 100	4 min 40 s 4.28× shorter Did not pass, 6.8 of 100

Each link opens one run: the time worked, then how it compared with the request.

Data and citation

Upstream: Yoran et al., 2024 (EMNLP)