Implement generative AI and agentic solutions 2.2 Build agents by using Foundry
A product team wants an evaluator that scores overall user satisfaction across whole…
Study guide skill: Integrate monitoring into deployed agents, evaluate agent behavior, and perform error analysis
Question B8-024
A product team wants an evaluator that scores overall user satisfaction across whole multi-turn support conversations with an agent and detects user frustration. Which agent evaluator fits?
Select one answer.
Answer: D
Customer Satisfaction (preview) measures holistic user satisfaction across a conversation on six dimensions and is used for detecting user frustration. Intent Resolution checks whether the agent identified the user's intent, Task Completion checks for a usable deliverable that meets the requirements, and Task Adherence checks that actions follow the system message's rules and constraints.
Deep dive
How it works
Foundry's agent evaluators include several system-level evaluators that sound similar but answer different questions. The one built for this team is Customer Satisfaction (preview):
- Use for: measuring overall user satisfaction across a conversation and detecting user frustration.
- What it measures: holistic user satisfaction across six dimensions: helpfulness, completeness, clarity, tone, resolution and adaptability.
- Output: a 1-5 Likert scale.
The neighbors:
- Intent Resolution (preview): whether the agent correctly identifies the user's intent.
- Task Completion (preview): whether the agent completed the requested task with a usable deliverable that meets all user requirements.
- Task Adherence (preview): whether the agent's actions follow its assigned tasks according to rules, procedures and policy constraints, based on its system message and prior steps.
The other options, and when you'd use them
- Intent Resolution: It checks that the agent understood what the user wanted. It doesn't score satisfaction or frustration.
- Task Completion: It checks for a usable deliverable that meets requirements. A task can be completed and the user still frustrated.
- Task Adherence: It checks compliance with the system message's rules and constraints.
Exam tip
Satisfaction or frustration over a whole conversation = Customer Satisfaction. Understood the ask = Intent Resolution. Delivered = Task Completion. Followed the rules = Task Adherence.
References
Last reviewed: