Implement information extraction solutions 5.1 Build retrieval and grounding pipelines
HR process PDFs explain approval steps only in embedded diagrams.
Study guide skill: Ingest and index content, such as documents, images, audio, and video
Question B2-027
HR process PDFs explain approval steps only in embedded diagrams. You're building a multimodal Azure AI Search index with a text embedding model and want the diagrams to be findable through text and vector search. Which skill should the pipeline use to describe each extracted image in natural language?
Select one answer.
Answer: B
In a multimodal pipeline, the GenAI Prompt skill uses an LLM to verbalize images, producing concise natural-language descriptions that are then embedded and searched like text. That lets a question be answered even when the only description of a process lives in an embedded diagram. Key phrase and language detection work on existing text, and semantic ranker reranks results at query time.
Deep dive
How it works
A multimodal Azure AI Search pipeline can make images searchable in two ways: image verbalization or direct multimodal embeddings. Because this index uses a text embedding model, verbalization is the path that fits.
With verbalization, the GenAI Prompt skill calls a large language model during ingestion to write a concise natural-language description of each extracted image. The docs give an example: "Five-step HR access workflow that begins with manager approval." That description is stored as text and embedded alongside the surrounding document text, using Azure OpenAI, Foundry or Azure Vision embedding models. Once the diagram is expressed in words, search can:
- Interpret the relationships and entities shown in the diagram
- Supply ready-made captions that an LLM can cite in its answer
- Return relevant snippets for RAG, even when the only explanation of a process is in a picture
The tradeoff is cost and time: this extra semantic depth means one LLM call for every image and a small increase in indexing time. The other path, direct multimodal embeddings, sends images and text to a multimodal embedding model that puts them in the same vector space, with no LLM call at indexing. That suits "find something that looks like this" searches, but it doesn't explain why two images are related or give the LLM text to cite.
The other options, and when you'd use them
- The Language Detection skill: It detects the language of input text and returns a language code with a score. It needs existing text and can't describe an image.
- Semantic ranker: It reranks search results at query time by using language understanding models. It improves ordering but adds nothing to the index, so diagrams without text still can't be found.
- The Key Phrase Extraction skill: It returns key phrases from unstructured text. With no text to start from, there's nothing to extract.
Exam tip
"Make diagrams searchable with a text embedding model" means verbalize the images with the GenAI Prompt skill. "Visual similarity, no LLM at indexing" means direct multimodal embeddings.
References
Last reviewed: