Ranked 8 of 9 on the combined score
Muse Wonder 1.0
Muse Code, max reasoning. 345 timed runs on 200 tasks from 18 benchmarks.
Duration following
Does it work for as long as it is asked? Each task was asked for a short and a long duration: a quarter and four times its own forecast of each task (on PaperBench and PostTrainBench, of its runtime with no time asked).
| Ask | Timing error | Where runs ended |
|---|---|---|
| Every timed run345 runs | 9.07× | |
| Short ask174 runs | 3.43× | |
| Long ask171 runs | 24.16× |
Its typical run was early at both asks: median 0.63× of the time asked at the short one, 0.04× at the long one. See every run
Every Muse Wonder run
Each row is one run, with a mark on the chart. Hover a row to find its mark; click a mark to find its row.
345 runs
| Loading 345 runs. | |||
No runs match these filters.
Forecasting
Before each task, it forecast how long the task would take. Each forecast is set against its own runtime on that task: its run with no time asked.
- Typical miss
- 8.44×
- Geometric mean of the factor between forecast and runtime
- Most forecasts
- 92% too high
- On 175 tasks
- Median forecast
- 40 min
- Median runtime
- 8 min
Retrospection
After its runs had finished, it was asked how long they took, with less time information at each step. Each figure is its typical miss against the real runtime, two answers per condition.
- Oracle
- 1.01×
- Native
- 1.65×
- Context-only
- 1.87×
- Replay
- 2.96×
- Scrubbed
- 3.58×
Oracle adds an elapsed-time tool. Native and Context-only continue the finished session, with tools on or off. Replay sends a rebuilt transcript to the model's API, and Scrubbed does the same without timestamps or other time cues. Its row covers 21 tasks.
Across the 18 benchmarks
Timing on its timed runs, and how good the work was. Score is the mean of its graded runs out of 100, from its runs with no time asked; the benchmark score is the mean of these, each benchmark counting the same.
| ProgramBench33 runs | 45.66× | 0% | 46.815 graded |
| PPTArena20 runs | 31.83× | 0% | Too few graded |
| YC-Bench4 runs | 31.49×(few runs) | 0% | Own units |
| Sakana ALE-Bench20 runs | 23.39× | 0% | Own units |
| Agents' Last Exam24 runs | 19.83× | 0% | 52.511 graded |
| TUA-Bench19 runs | 11.94× | 0% | 66.210 graded |
| METR public tasks3 runs | 11.91×(few runs) | 0% | Too few graded |
| AppWorld28 runs | 11.50× | 7% | 50.014 graded |
| DeepSWE28 runs | 9.05× | 0% | 69.213 graded |
| CORE-Bench24 runs | 7.46× | 0% | 91.712 graded |
| Terminal-Bench 4.032 runs | 7.38× | 6% | 62.516 graded |
| OSWorld6 runs | 4.25×(few runs) | 0% | 35.58 graded |
| BrowseComp24 runs | 2.77× | 21% | 75.012 graded |
| GPQA Diamond24 runs | 2.64× | 0% | 83.312 graded |
| Humanity's Last Exam40 runs | 2.44× | 5% | 35.020 graded |
| AssistantBench16 runs | 1.61× | 31% | 17.38 graded |
No runs match these filters.
Highest timing error first. ALE-Bench and YC-Bench keep their own units and stay out of the score, as do PaperBench and PostTrainBench.
Read before quoting. 1 run was stopped by AgentTime, so its time is a lower bound. Details