Ranked 8 of 9 on the combined score

Muse Wonder 1.0

Muse Code, max reasoning. 345 timed runs on 200 tasks from 18 benchmarks.

14%Combined scoreOver the three tests
5%Within 5%76% shorter, 19% longer
8.44×Forecasting missIts typical forecast against its runtime
1.65×Retrospection missNative, its typical estimate of a finished run
57.1Benchmark scoreOut of 100, 95% interval 50.4 to 63.5

Duration following

Does it work for as long as it is asked? Each task was asked for a short and a long duration: a quarter and four times its own forecast of each task (on PaperBench and PostTrainBench, of its runtime with no time asked).

Where Muse Wonder 1.0's runs ended, from a tenth of the time asked to ten times it, for every run and for each ask, with the timing error of each.
AskTiming errorWhere runs ended
Every timed run345 runs9.07×
Short ask174 runs3.43×
Long ask171 runs24.16×

Its typical run was early at both asks: median 0.63× of the time asked at the short one, 0.04× at the long one. See every run

Every Muse Wonder run

Each row is one run, with a mark on the chart. Hover a row to find its mark; click a mark to find its row.

Ask
Result

345 runs

Muse Wonder 1.0's runs. Each row links to its run page.
Loading 345 runs.

Forecasting

Before each task, it forecast how long the task would take. Each forecast is set against its own runtime on that task: its run with no time asked.

Typical miss
8.44×
Geometric mean of the factor between forecast and runtime
Most forecasts
92% too high
On 175 tasks
Median forecast
40 min
Median runtime
8 min

Retrospection

After its runs had finished, it was asked how long they took, with less time information at each step. Each figure is its typical miss against the real runtime, two answers per condition.

Oracle
1.01×
Native
1.65×
Context-only
1.87×
Replay
2.96×
Scrubbed
3.58×

Oracle adds an elapsed-time tool. Native and Context-only continue the finished session, with tools on or off. Replay sends a rebuilt transcript to the model's API, and Scrubbed does the same without timestamps or other time cues. Its row covers 21 tasks.

Across the 18 benchmarks

Timing on its timed runs, and how good the work was. Score is the mean of its graded runs out of 100, from its runs with no time asked; the benchmark score is the mean of these, each benchmark counting the same.

Muse Wonder 1.0 across the 18 benchmarks
ProgramBench33 runs
45.66×
0%46.815 graded
PPTArena20 runs
31.83×
0%Too few graded
YC-Bench4 runs
31.49×(few runs)
0%Own units
Sakana ALE-Bench20 runs
23.39×
0%Own units
Agents' Last Exam24 runs
19.83×
0%52.511 graded
TUA-Bench19 runs
11.94×
0%66.210 graded
METR public tasks3 runs
11.91×(few runs)
0%Too few graded
AppWorld28 runs
11.50×
7%50.014 graded
DeepSWE28 runs
9.05×
0%69.213 graded
CORE-Bench24 runs
7.46×
0%91.712 graded
Terminal-Bench 4.032 runs
7.38×
6%62.516 graded
OSWorld6 runs
4.25×(few runs)
0%35.58 graded
BrowseComp24 runs
2.77×
21%75.012 graded
GPQA Diamond24 runs
2.64×
0%83.312 graded
Humanity's Last Exam40 runs
2.44×
5%35.020 graded
AssistantBench16 runs
1.61×
31%17.38 graded

Highest timing error first. ALE-Bench and YC-Bench keep their own units and stay out of the score, as do PaperBench and PostTrainBench.

Read before quoting. 1 run was stopped by AgentTime, so its time is a lower bound. Details

Data and citation