XML fence guard against Role then format

Both scored on the same cases against nemotron-3-super-120b-a12b, same judge. The intervals tell you whether the gap is real or noise.

suite structured-json-extractionmodel nemotron-3-super-120b-a12bn=18 public cases each

xml-fence-guard

v2 by @dara

0.0

0.0 to 17.6 · holdout 0.0

v2 by @ilya

27.8

12.5 to 50.9 · holdout 16.7

Intervals overlap. The 27.8 point gap is not significant at this sample size.

Metricxml-fence-guardrole-then-formatBetter

Public score0.027.8right

Holdout score0.016.7right

Interval width17.638.4left

Cost per run$0.000$0.000same

Median latency6.5s6.7sleft

Sample size1818same

Overlap: Two entries are only meaningfully apart when their confidence intervals do not overlap. A higher number with an overlapping interval is a coin flip, not a win.
Holdout: The public score can be gamed by tuning to visible cases. A gap between public and holdout is the tell.
Reproduce: evalness compare \ xml-fence-guard role-then-format \ --model nemotron-3-super-120b-a12b