Task 15 of 18 in AppWorld
Send individualized acceptance or rejection replies using local decision templates
Astra's three runs landed closest to the request (2 min 45 s, 10 min 41 s and 40 min 58 s). Fable worked 5 min 13 s, 4 min 3 s and 5 min 50 s whatever it was asked.
- Asked for
- 2.5, 10 and 40 min
- Runs here
- 9 of 9
One task, asked three ways
- Astra
- Sol
- Fable
One line per agent joins its three runs.
What we asked
Please work on this task for a full [2.5 | 10 | 40] minutes. Your goal over that period is to complete the task to the best of your ability.
One sentence, sent three times with a different time: 2.5, 10 and 40 min.
The three sentences in full
Shortest2.5 minutes
Please work on this task for a full 2.5 minutes. Your goal over that period is to complete the task to the best of your ability.
Middle10 minutes
Please work on this task for a full 10 minutes. Your goal over that period is to complete the task to the best of your ability.
Longest40 minutes
Please work on this task for a full 40 minutes. Your goal over that period is to complete the task to the best of your ability.
Every run on this task
| Agent | Shortest | Middle | Longest |
|---|---|---|---|
| Astra | 2 min 45 s 1.10× longer Passed, 100 of 100 | 10 min 41 s 1.07× longer Passed, 100 of 100 | 40 min 58 s Within 5% Passed, 100 of 100 |
| Sol | 2 min 52 s 1.15× longer Passed, 100 of 100 | 11 min 1.10× longer Passed, 100 of 100 | 41 min 29 s Within 5% Passed, 100 of 100 |
| Fable | 5 min 13 s 2.09× longer Passed, 100 of 100 | 4 min 3 s 2.47× shorter Passed, 100 of 100 | 5 min 50 s 6.86× shorter Passed, 100 of 100 |
No runs match these filters.
Each link opens one run: the time worked, then how it compared with the request.
Data and citation
Upstream: Trivedi et al., 2024 (ACL)