registry/compare
First tool guess against Role then format
Both scored on the same cases against claude-fable-5, same judge. The intervals tell you whether the gap is real or noise.
suite reasoning-chainmodel claude-fable-5n=154 public cases each
first-tool-guess
v1 by @system75.3
68.0 to 81.5 · holdout 74.2
role-then-format
v2 by @ilya91.7
86.5 to 95.0 · holdout 90.3
Intervals do not overlap. The 16.4 point gap is significant at this sample size.role-then-format wins
Metricfirst-tool-guessrole-then-formatBetter
Public score75.391.7right
Holdout score74.290.3right
Interval width13.58.5right
Cost per run$1.814$1.850left
Median latency1.4s2.4sleft
Sample size154168right
Reading this verdict
- Overlap
- Two entries are only meaningfully apart when their confidence intervals do not overlap. A higher number with an overlapping interval is a coin flip, not a win.
- Holdout
- The public score can be gamed by tuning to visible cases. A gap between public and holdout is the tell.
- Reproduce
- evalness compare \ first-tool-guess role-then-format \ --model claude-fable-5