Plan and manage an Azure AI solution 1.4 Implement responsible AI across generative AI and agentic systems
A team needs two custom evaluators (preview) in Foundry.
Study guide skill: Apply responsible AI instrumentation, including evaluators, safety evaluations, and explanation tooling
Question B6-011
A team needs two custom evaluators (preview) in Foundry. The first must check deterministically that every response contains a required disclaimer and stays under a word limit. The second must judge whether responses match the brand's tone. Which evaluator types should they create?
Select one answer.
Answer: A
Code-based evaluators use a Python grade() function with deterministic logic and suit rule-based checks, format validation, and length limits; they return a float from 0.0 to 1.0. Prompt-based evaluators use a judge prompt and suit subjective judgments such as tone analysis. Endpoint-based evaluators are one of three types, for scoring logic hosted on your own infrastructure, not a requirement for every custom evaluator.
Deep dive
How it works
Foundry lets you create three types of custom evaluators (preview), and each fits a different kind of check:
- Code-based: a Python
grade()function scores each item with deterministic logic. Best for rule-based checks, keyword matching, format validation and length limits. It returns a single float from 0.0 to 1.0, where higher is better. - Prompt-based: a judge prompt tells an LLM how to score each item. Best for subjective quality judgments, semantic similarity and tone analysis. Scoring can be ordinal, continuous or binary, with a range you define.
- Endpoint-based: an external HTTP endpoint receives evaluation data and returns scores. Best for custom scoring logic hosted on your own infrastructure, proprietary models, or pipelines that need network access.
So the disclaimer-plus-word-limit check is code-based (a string check and a word count, exactly repeatable), and brand tone is prompt-based. Microsoft names brand tone as a typical reason to build a custom evaluator.
One detail worth knowing: if grade() raises an exception or times out, that item is recorded as 0.0 and marked as an error in the evaluation report.
The other options, and when you'd use them
- Prompt-based for the rules, code-based for tone: Reversed. Length limits and format checks are listed under code-based; tone analysis is listed under prompt-based.
- Endpoint-based for both: Endpoint-based is one of three types, for scoring hosted on your own infrastructure. Custom evaluators don't have to call an endpoint.
- Code-based returns only Pass/Fail: It returns a continuous float from 0.0 to 1.0.
Exam tip
Deterministic rule = code-based (grade(), 0.0 to 1.0). Subjective judgment = prompt-based. Your own service = endpoint-based.
References
Last reviewed: