Implement generative AI and agentic solutions 2.2 Build agents by using Foundry
Traces show that an agent sometimes calls the right tool with wrong parameter values.
Study guide skill: Integrate monitoring into deployed agents, evaluate agent behavior, and perform error analysis
Question B1-030
Traces show that an agent sometimes calls the right tool with wrong parameter values. Which agent evaluator directly measures whether the agent made the right tool calls with correct parameters?
Select one answer.
Answer: C
Tool Call Accuracy measures whether the agent made the right tool calls with correct parameters to complete its task. Intent Resolution checks whether the user's intent was identified, Task Adherence checks compliance with assigned rules and procedures, and Coherence is a general text-quality metric.
Deep dive
How it works
Foundry's agent evaluators split into two groups. System evaluation looks at the quality of the final outcome. Process evaluation checks the step-by-step execution the agent took to get there. A wrong parameter value on a tool call is a process problem, so you want a process evaluator.
Tool Call Accuracy is the one the question describes. It measures whether the agent made the right tool calls with correct parameters to complete its task, and it returns Pass/Fail based on a threshold on a 1-5 scale. The docs position it as an overall tool-call quality check for agents that use tools and APIs.
Related process evaluators help narrow a problem down:
- Tool Selection: did the agent pick the correct tools without unnecessary ones?
- Tool Input Accuracy: are all tool-call parameters correct across six strict criteria? This is for strict parameter validation in production or API integration tests.
- Tool Use Quality (preview): batches Tool Call Accuracy, Tool Call Success, Tool Input Accuracy, Tool Output Utilization and Tool Selection into one LLM judge call to cut evaluation cost and latency.
Running these on the traces where the wrong values showed up tells you whether the agent picked the wrong tool or just filled in the right tool badly.
The other options, and when you'd use them
- Intent Resolution: A system evaluator that measures whether the agent correctly identified the user's intent. An agent can understand the request perfectly and still pass wrong parameters.
- Coherence: A general-purpose quality evaluator for the logical, orderly presentation of ideas in a response. It says nothing about tool calls.
- Task Adherence: A system evaluator that checks whether the agent's actions follow its assigned tasks, rules and policy constraints from its system message. It's about compliance, not parameter correctness.
Exam tip
Match the symptom to the evaluator's name: "right tool, wrong parameters" means Tool Call Accuracy (or Tool Input Accuracy for strict validation), "wrong tool" means Tool Selection, and "misunderstood the user" means Intent Resolution.
References
Last reviewed: