registrytool-call-accuracysignature-echo

Signature echo

Echoes the chosen tool signature before emitting arguments. Catches argument name drift on tools with similar shapes.

Compare

Prompt

// system Choose one tool. Restate its full signature, then emit the call with arguments filled from the request. // user Tools: {{tools}} Request: {{request}}

Score history

Model versionVersionScore · CIHoldoutState
llama-4.2-405bv278.6 71.4–84.377.3fresh
gemini-3.2-prov286.4 80.1–90.986.4fresh
claude-fable-5v292.9 87.7–96.092.4fresh
claude-opus-4.8v289.6 83.8–93.587.9fresh
mistral-large-3v282.5 75.7–87.781.8fresh
gpt-5.6-solv290.9 85.3–94.590.9fresh
claude-fable-4v184.4 77.9–89.383.3superseded
current scorev2
78.6
95% CI 71.4 to 84.3 · n=154
Holdout score77.3
Judgeprogrammatic
Cost per run$0.259
Median latency2.6s
LicenseCC-BY-4.0
Author@dara
0 endorsements

Sign in to endorse this entry.

Across models

llama-4.2-405b78.6fresh
gemini-3.2-pro86.4fresh
claude-fable-592.9fresh
claude-opus-4.889.6fresh
mistral-large-382.5fresh
gpt-5.6-sol90.9fresh

Versions

v2Added signature restatement78.6
v1First submission84.4