registry/compare
Role then format against Decompose into subgoals
Both scored on the same cases against claude-fable-5, same judge. The intervals tell you whether the gap is real or noise.
suite reasoning-chainmodel claude-fable-5n=168 public cases each
role-then-format
v2 by @ilya91.7
86.5 to 95.0 · holdout 90.3
decompose-subgoals
v1 by @ilya85.7
78.5 to 90.8 · holdout 85.2
Intervals overlap. The 6.0 point gap is not significant at this sample size.
Metricrole-then-formatdecompose-subgoalsBetter
Public score91.785.7left
Holdout score90.385.2left
Interval width8.512.3left
Cost per run$1.850$0.987right
Median latency2.4s2.6sleft
Sample size168126left
Reading this verdict
- Overlap
- Two entries are only meaningfully apart when their confidence intervals do not overlap. A higher number with an overlapping interval is a coin flip, not a win.
- Holdout
- The public score can be gamed by tuning to visible cases. A gap between public and holdout is the tell.
- Reproduce
- evalness compare \ role-then-format decompose-subgoals \ --model claude-fable-5