Implement generative AI and agentic solutions 2.2 Build agents by using Foundry
Traces of an inventory agent show that it calls the right stock-lookup tool with correct…
Study guide skill: Integrate monitoring into deployed agents, evaluate agent behavior, and perform error analysis
Question B3-017
Traces of an inventory agent show that it calls the right stock-lookup tool with correct parameters, and the tool returns correct quantities, yet the agent's final answer sometimes states the wrong quantity. Which agent evaluator directly measures this failure?
Select one answer.
Answer: C
Tool Output Utilization checks whether the agent correctly understood and used tool results in its reasoning and final response, which is exactly where this agent fails. Tool Call Accuracy measures whether the agent made the right calls with correct parameters, and Tool Selection whether it chose the right tools, and both are already fine here. Intent Resolution (preview) measures whether the agent identified the user's intent.
Deep dive
How it works
Foundry's agent evaluators split tool use into separate steps, so you can pin down exactly where an agent goes wrong:
- Tool Selection: did the agent pick the correct tools, without unnecessary ones?
- Tool Call Accuracy: did it make the right tool calls with correct parameters?
- Tool Call Success: did the calls succeed without technical errors?
- Tool Output Utilization: did it correctly understand and use the tool call results in its reasoning and final response?
The traces show the first three are fine: right tool, right parameters, correct results. The failure is after the tool returns: the final answer misstates what the tool said. That's precisely what Tool Output Utilization measures, and it returns Pass/Fail.
If you want one judge call that covers the whole tool-use process, Tool Use Quality (preview) batches these tool evaluators, but the one that targets this failure directly is Tool Output Utilization.
The other options, and when you'd use them
- Tool Call Accuracy: It checks the calls and their parameters, which the traces already show are correct.
- Intent Resolution (preview): It measures whether the agent correctly identifies the user's intent. The agent understood the question; it misused the answer.
- Tool Selection: It checks that the agent chose the correct tools, which it did.
Exam tip
Wrong tool = Tool Selection. Wrong parameters = Tool Call Accuracy. Right data, wrong answer = Tool Output Utilization.
References
Last reviewed: