Ranked 1 of 9 on the combined score
GPT 6 Astra
Codex, max reasoning. 400 timed runs on 200 tasks from 18 benchmarks.
Duration following
Does it work for as long as it is asked? Each task was asked for a short and a long duration: the paper’s shortest and longest requests, about a quarter and four times each task’s usual time.
| Ask | Timing error | Where runs ended |
|---|---|---|
| Every timed run400 runs | 1.16× | |
| Short ask200 runs | 1.18× | |
| Long ask200 runs | 1.14× |
It ran long on short asks (median 1.11× at the short ask) and was within 5% of the time asked on long ones (median 1.01× at the long ask). See every run
Every Astra run
Each row is one run, with a mark on the chart. Hover a row to find its mark; click a mark to find its row.
400 runs
| Loading 400 runs. | |||
No runs match these filters.
Forecasting
Before each task, it forecast how long the task would take. Each forecast is set against its own runtime on that task: the paper’s estimate from its timed runs.
- Typical miss
- 2.96×
- Geometric mean of the factor between forecast and runtime
- Most forecasts
- 63% too high
- On 200 tasks
- Median forecast
- 20 min
- Median runtime
- 11 min
Retrospection
After its runs had finished, it was asked how long they took, with less time information at each step. Each figure is its typical miss against the real runtime, two answers per condition.
- Oracle
- 1.00×
- Native
- 1.10×
- Context-only
- 1.22×
- Replay
- 1.20×
- Scrubbed
- 1.39×
Oracle adds an elapsed-time tool. Native and Context-only continue the finished session, with tools on or off. Replay sends a rebuilt transcript to the model's API, and Scrubbed does the same without timestamps or other time cues. On 23 tasks.
Across the 18 benchmarks
Timing on its timed runs, and how good the work was. Score is the mean of its graded runs out of 100, from its middle request, about the task’s usual time; the benchmark score is the mean of these, each benchmark counting the same.
| YC-Bench6 runs | 2.22×(few runs) | 17% | Own units |
| AssistantBench16 runs | 1.70× | 19% | 3.68 graded |
| PPTArena20 runs | 1.36× | 45% | 63.010 graded |
| OSWorld32 runs | 1.20× | 53% | 67.316 graded |
| Agents' Last Exam24 runs | 1.14× | 67% | 56.412 graded |
| TUA-Bench20 runs | 1.13× | 50% | 92.010 graded |
| AppWorld36 runs | 1.09× | 50% | 66.718 graded |
| Humanity's Last Exam48 runs | 1.09× | 50% | 79.224 graded |
| DeepSWE24 runs | 1.08× | 58% | 83.312 graded |
| GPQA Diamond56 runs | 1.07× | 48% | 96.428 graded |
| Terminal-Bench 4.032 runs | 1.05× | 84% | 56.316 graded |
| CORE-Bench24 runs | 1.04× | 63% | 33.312 graded |
| Sakana ALE-Bench20 runs | 1.02× | 95% | Own units |
| ProgramBench28 runs | 1.01× | 100% | 77.314 graded |
| PaperBench6 runs | 1.01×(few runs) | 100% | Not scored |
| PostTrainBench4 runs | 1.01×(few runs) | 100% | Not scored |
| METR public tasks4 runs | 1.00×(few runs) | 100% | Too few graded |
No runs match these filters.
Highest timing error first. ALE-Bench and YC-Bench keep their own units and stay out of the score, as do PaperBench and PostTrainBench.
Read before quoting. 8 runs were stopped by AgentTime, so their time is a lower bound. Details