Plan and manage an Azure AI solution 1.3 Manage, monitor, and secure AI systems
A RAG app's answers are getting worse, and you suspect the search step.
Study guide skill: Monitor data ingestion quality, search index health, and relevance performance
Question B4-011
A RAG app's answers are getting worse, and you suspect the search step. For a test set of queries you have relevance labels (qrels) saying which documents each query should return, and you want ranking metrics such as NDCG to tune search parameters. Which evaluator should you use?
Select one answer.
Answer: C
Document Retrieval is the process evaluator for when retrieval is the bottleneck and you have query relevance labels; it compares retrieved documents with the labels and reports metrics such as Fidelity, NDCG, and XDCG. Retrieval uses an LLM judge when you don't have ground truth. Groundedness and Response Completeness score the generated response, not the search results.
Deep dive
How it works
Foundry splits RAG evaluation into process evaluators (the retrieval step) and system evaluators (the final response). When you suspect the search step, you want a process evaluator, and there are two:
- Retrieval: an LLM judge scores how relevant the retrieved context chunks are to the query. Use it when you don't have ground truth. Inputs: query and context, plus a deployment_name for the judge.
- Document Retrieval: for when retrieval quality is a bottleneck and you have query relevance labels (qrels). It compares retrieved documents with the labels and reports search quality metrics: Fidelity, NDCG, XDCG, Max Relevance and Holes. Inputs: retrieval_ground_truth and retrieved_documents, with no judge model needed.
You have qrels and want NDCG to tune search parameters, which is the Document Retrieval use case word for word.
The other options, and when you'd use them
- Groundedness: A system evaluator that checks whether the response stays within the context without fabricating content. It scores the answer, not the search results.
- Retrieval: It's for when you don't have ground truth and want an LLM judge's view of chunk relevance. It doesn't compute ranking metrics against your labels.
- Response Completeness (preview): A system evaluator that checks whether the response covers the ground-truth answer (recall). It also scores the answer, not ranking.
Exam tip
Search quality with qrels and NDCG = Document Retrieval. Search quality without labels = Retrieval (LLM judge).
References
Last reviewed: