registrycompare

Role then format against Few shot triplet

Both scored on the same cases against claude-fable-5, same judge. The intervals tell you whether the gap is real or noise.

Swap sides
suite reasoning-chainmodel claude-fable-5n=168 public cases each

role-then-format

v2 by @ilya
91.7
86.5 to 95.0 · holdout 90.3

few-shot-triplet

v1 by @nils
84.5
78.3 to 89.2 · holdout 83.3
Intervals overlap. The 7.2 point gap is not significant at this sample size.
Metricrole-then-formatfew-shot-tripletBetter
Public score91.784.5left
Holdout score90.383.3left
Interval width8.510.9left
Cost per run$1.850$1.850same
Median latency2.4s2.7sleft
Sample size168168same

Reading this verdict

Overlap
Two entries are only meaningfully apart when their confidence intervals do not overlap. A higher number with an overlapping interval is a coin flip, not a win.
Holdout
The public score can be gamed by tuning to visible cases. A gap between public and holdout is the tell.
Reproduce
evalness compare \ role-then-format few-shot-triplet \ --model claude-fable-5