Methodology

Scores

A score is the share of cases an entry passed, on one suite, against one named model version, at one point in time. It is reported with a 95 percent Wilson confidence interval, which behaves correctly near 0 and 100 percent where the normal approximation produces impossible bounds.

Two entries are only meaningfully different when their intervals do not overlap. The compare page states this verdict explicitly rather than leaving you to eyeball two numbers.

Holdout

Thirty percent of every suite is withheld and never published. Assignment is deterministic from the case key, so republishing a suite cannot leak a withheld case, and the same case always lands on the same side of the split.

Both the public and holdout score are shown. A large gap between them means the entry was tuned to the visible cases and will not generalise.

Decay

A score belongs to one model version. When a provider ships a new version, the old score is frozen, not migrated, and the entry is marked stale until it is rerun. Retired versions keep their scores forever.

Immutability

Prompts, cases, per-case results, and published scores are insert-only. This is enforced by database triggers, not by application code, so a console session or a future migration cannot quietly rewrite history. A correction is a new row; the old one is marked superseded and stays visible.

Cost

Every run reports the spend it actually incurred, computed from token counts and the provider rate for that version. Three caps apply: per run, per user per day, and a global daily ceiling. A run that would cross any of them is refused before it starts.

What this does not tell you

A suite is a proxy, not the truth. A high score means an entry did well on these cases with this judge, not that it is the best prompt for your task. The cases are published so you can decide whether the proxy resembles your problem.