Implement generative AI and agentic solutions 2.2 Build agents by using Foundry
Traces show that a production agent's tool calls sometimes fail with timeouts and API…
Study guide skill: Integrate monitoring into deployed agents, evaluate agent behavior, and perform error analysis
Question B6-024
Traces show that a production agent's tool calls sometimes fail with timeouts and API exceptions. You want an evaluator that tracks this tool reliability over time. Which agent evaluator fits?
Select one answer.
Answer: A
Tool Call Success measures whether tool calls succeeded or resulted in technical errors or exceptions, and it's meant for monitoring tool reliability, API failures, and timeouts. Tool Input Accuracy checks parameters against strict criteria, and Tool Output Utilization checks whether the agent used tool results correctly. Tool Call Accuracy measures whether the agent made the right tool calls with correct parameters.
Deep dive
How it works
Foundry's agent evaluators split tool behavior into separate questions. The one that matches "did the call itself work?" is Tool Call Success:
- What it measures: whether tool calls succeeded or resulted in technical errors or exceptions.
- What it's for: monitoring tool reliability, detecting API failures, timeout issues or technical errors in tool execution.
- Output: binary Pass/Fail.
Timeouts and API exceptions seen in traces are exactly that, and running the evaluator over time gives you the reliability trend.
The neighboring evaluators look at whether the agent used tools correctly, not whether the tools worked:
- Tool Call Accuracy: whether the agent made the right tool calls with correct parameters to complete its task.
- Tool Input Accuracy: strict validation of tool parameters, for workflows that need 100% parameter correctness.
- Tool Output Utilization: whether the agent correctly understood and used tool call results in its reasoning and final response.
The other options, and when you'd use them
- Tool Output Utilization: It's about what the agent does with a result it got. A timeout gives it no result to use.
- Tool Call Accuracy: It judges choosing the right tools and parameters. A correct call can still time out.
- Tool Input Accuracy: It's for strict parameter validation, not execution failures.
Exam tip
Errors, exceptions, timeouts = Tool Call Success. Right tool and params = Tool Call Accuracy. Params strictly correct = Tool Input Accuracy. Used the result = Tool Output Utilization.
References
Last reviewed: