Ranked 4 of 9 on the combined score

Claude Fable 5.1

Claude Code, max reasoning. 395 timed runs on 200 tasks from 18 benchmarks.

26%Combined scoreOver the three tests
4%Within 5%53% shorter, 43% longer
2.47×Forecasting missIts typical forecast against its runtime
1.32×Retrospection missNative, its typical estimate of a finished run
60.0Benchmark scoreOut of 100, 95% interval 54.6 to 64.6

Duration following

Does it work for as long as it is asked? Each task was asked for a short and a long duration: the paper’s shortest and longest requests, about a quarter and four times each task’s usual time.

Where Claude Fable 5.1's runs ended, from a tenth of the time asked to ten times it, for every run and for each ask, with the timing error of each.
AskTiming errorWhere runs ended
Every timed run395 runs3.37×
Short ask199 runs3.24×
Long ask196 runs3.34×

It ran long on short asks (median 1.77× at the short ask) and stopped early on long ones (median 0.29× at the long ask). See every run

Every Fable run

Each row is one run, with a mark on the chart. Hover a row to find its mark; click a mark to find its row.

Ask
Result

395 runs

Claude Fable 5.1's runs. Each row links to its run page.
Loading 395 runs.

Forecasting

Before each task, it forecast how long the task would take. Each forecast is set against its own runtime on that task: the paper’s estimate from its timed runs.

Typical miss
2.47×
Geometric mean of the factor between forecast and runtime
Most forecasts
63% too high
On 192 tasks
Median forecast
15 min
Median runtime
14 min

Retrospection

After its runs had finished, it was asked how long they took, with less time information at each step. Each figure is its typical miss against the real runtime, two answers per condition.

Oracle
1.00×
Native
1.32×
Context-only
1.63×
Replay
1.55×
Scrubbed
1.51×

Oracle adds an elapsed-time tool. Native and Context-only continue the finished session, with tools on or off. Replay sends a rebuilt transcript to the model's API, and Scrubbed does the same without timestamps or other time cues. On 23 tasks.

Across the 18 benchmarks

Timing on its timed runs, and how good the work was. Score is the mean of its graded runs out of 100, from its middle request, about the task’s usual time; the benchmark score is the mean of these, each benchmark counting the same.

Claude Fable 5.1 across the 18 benchmarks
PostTrainBench3 runs
6.60×(few runs)
0%Not scored
METR public tasks4 runs
5.04×(few runs)
0%Too few graded
GPQA Diamond56 runs
4.05×
5%85.728 graded
Agents' Last Exam24 runs
3.71×
0%57.312 graded
AppWorld36 runs
3.63×
0%72.218 graded
AssistantBench16 runs
3.57×
0%2.68 graded
TUA-Bench20 runs
3.52×
5%73.910 graded
DeepSWE24 runs
3.49×
0%83.312 graded
PPTArena20 runs
3.44×
5%45.010 graded
YC-Bench6 runs
3.43×(few runs)
0%Own units
CORE-Bench24 runs
3.33×
4%0.012 graded
Terminal-Bench 4.032 runs
3.15×
0%68.816 graded
Humanity's Last Exam48 runs
3.04×
6%87.524 graded
PaperBench6 runs
2.63×(few runs)
0%Not scored
ProgramBench24 runs
2.62×
0%67.212 graded
OSWorld32 runs
2.30×
19%76.216 graded
Sakana ALE-Bench20 runs
1.95×
10%Own units

Highest timing error first. ALE-Bench and YC-Bench keep their own units and stay out of the score, as do PaperBench and PostTrainBench.

Read before quoting. 35 runs were stopped by AgentTime, so their time is a lower bound. Details

Data and citation