Role then format against Plain instruction

Both scored on the same cases against nemotron-3-super-120b-a12b, same judge. The intervals tell you whether the gap is real or noise.

suite structured-json-extractionmodel nemotron-3-super-120b-a12bn=18 public cases each

role-then-format

v2 by @ilya

27.8

12.5 to 50.9 · holdout 16.7

v1 by @system

0.0

0.0 to 17.6 · holdout 0.0

Intervals overlap. The 27.8 point gap is not significant at this sample size.

Metricrole-then-formatplain-instructionBetter

Public score27.80.0left

Holdout score16.70.0left

Interval width38.417.6right

Cost per run$0.000$0.000same

Median latency6.7s6.8sleft

Sample size1818same

Overlap: Two entries are only meaningfully apart when their confidence intervals do not overlap. A higher number with an overlapping interval is a coin flip, not a win.
Holdout: The public score can be gamed by tuning to visible cases. A gap between public and holdout is the tell.
Reproduce: evalness compare \ role-then-format plain-instruction \ --model nemotron-3-super-120b-a12b