Role then format against Few shot triplet

Both scored on the same cases against deepseek-v4-flash-0731, same judge. The intervals tell you whether the gap is real or noise.

suite structured-json-extractionmodel deepseek-v4-flash-0731n=18 public cases each

role-then-format

v2 by @ilya

27.8

12.5 to 50.9 · holdout 16.7

v1 by @nils

83.3

60.8 to 94.2 · holdout 83.3

Intervals do not overlap. The 55.5 point gap is significant at this sample size.few-shot-triplet wins

Metricrole-then-formatfew-shot-tripletBetter

Public score27.883.3right

Holdout score16.783.3right

Interval width38.433.4right

Cost per run$0.000$0.000same

Median latency6.8s6.8sleft

Sample size1818same

Overlap: Two entries are only meaningfully apart when their confidence intervals do not overlap. A higher number with an overlapping interval is a coin flip, not a win.
Holdout: The public score can be gamed by tuning to visible cases. A gap between public and holdout is the tell.
Reproduce: evalness compare \ role-then-format few-shot-triplet \ --model deepseek-v4-flash-0731