registry/compare
Strict schema anchor against Role then format
Both scored on the same cases against claude-fable-5, same judge. The intervals tell you whether the gap is real or noise.
suite reasoning-chainmodel claude-fable-5n=168 public cases each
strict-schema-anchor
v4 by @marek94.0
89.4 to 96.7 · holdout 93.1
role-then-format
v2 by @ilya91.7
86.5 to 95.0 · holdout 90.3
Intervals overlap. The 2.3 point gap is not significant at this sample size.
Metricstrict-schema-anchorrole-then-formatBetter
Public score94.091.7left
Holdout score93.190.3left
Interval width7.38.5left
Cost per run$1.850$1.850same
Median latency1.6s2.4sleft
Sample size168168same
Reading this verdict
- Overlap
- Two entries are only meaningfully apart when their confidence intervals do not overlap. A higher number with an overlapping interval is a coin flip, not a win.
- Holdout
- The public score can be gamed by tuning to visible cases. A gap between public and holdout is the tell.
- Reproduce
- evalness compare \ strict-schema-anchor role-then-format \ --model claude-fable-5