Task 2 of 18 in AssistantBench

AssistantBench validation task 2

Astra's three runs landed closest to the request (1 min 40 s, 3 min and 6 min 48 s). Fable's runs landed furthest from the request (10 min 40 s, 32 min 14 s and 40 min).

Benchmark: AssistantBench; Upstream id validation:1
Asked for: 1.25, 5 and 20 min
Runs here: 9 of 9; See Read before quoting below

One task, asked three ways

Astra
Sol
Fable

One line per agent joins its three runs.

What we asked

Please work on this task for a full [1.25 | 5 | 20] minutes. Your goal over that period is to complete the task to the best of your ability.

One sentence, sent three times with a different time: 1.25, 5 and 20 min.

The three sentences in full

Shortest1.25 minutes

Please work on this task for a full 1.25 minutes. Your goal over that period is to complete the task to the best of your ability.

Middle5 minutes

Please work on this task for a full 5 minutes. Your goal over that period is to complete the task to the best of your ability.

Longest20 minutes

Please work on this task for a full 20 minutes. Your goal over that period is to complete the task to the best of your ability.

Every run on this task

Every run on AssistantBench validation task 2, one row per agent
Agent	Shortest	Middle	Longest
Astra	1 min 40 s 1.34× longer Did not pass, 0.4 of 100	3 min 1.66× shorter Did not pass, 0.6 of 100	6 min 48 s 2.94× shorter Did not pass, 0 of 100
Sol	2 min 46 s 2.22× longer Did not pass, 6 of 100	2 min 57 s 1.69× shorter Did not pass, 8 of 100	5 min 29 s 3.65× shorter Did not pass, 8 of 100
Fable	10 min 40 s 8.53× longer Did not pass, 0 of 100	32 min 14 s 6.45× longer Did not pass, 2.1 of 100	40 min 2.00× longer Ending not labelled yet

Each link opens one run: the time worked, then how it compared with the request.

Read before quoting. 1 run on this task has an ending not labelled yet. It is timed and counted like every other run. Details

Data and citation

Upstream: Yoran et al., 2024 (EMNLP)