Ask when ambiguous
Emits a clarifying question instead of guessing when required arguments are missing. Scores well because a wrong call costs more than a question.
Prompt
// system
If any required argument is not determinable from the request, ask one clarifying question instead of calling a tool.
// user
Tools: {{tools}}
Request: {{request}}
Score history
Model versionVersionScore · CIHoldoutState
claude-opus-4.8v187.7 81.5–92.086.4fresh
mistral-large-3v180.5 73.6–86.080.3fresh
gemini-3.2-prov185.1 78.6–89.884.8fresh
claude-fable-5v190.3 84.6–94.089.4fresh
gpt-5.6-solv189.6 83.8–93.589.4fresh
llama-4.2-405bv176.6 69.3–82.675.8fresh
current scorev1
87.7
95% CI 81.5 to 92.0 · n=154
Holdout score86.4
Judgeprogrammatic
Cost per run$1.850
Median latency2.2s
LicenseCC-BY-4.0
Author@nils
0 endorsements
Sign in to endorse this entry.
Across models
claude-opus-4.887.7fresh
mistral-large-380.5fresh
gemini-3.2-pro85.1fresh
claude-fable-590.3fresh
gpt-5.6-sol89.6fresh
llama-4.2-405b76.6fresh
Versions
v1First submission87.7