Implement information extraction solutions 5.1 Build retrieval and grounding pipelines
A retailer indexes product photos in Azure AI Search.
Study guide skill: Ingest and index content, such as documents, images, audio, and video
Question B5-026
A retailer indexes product photos in Azure AI Search. Shoppers upload a photo and want visually similar products, and the team doesn't want to call an LLM for every image during indexing. Which approach fits?
Select one answer.
Answer: B
Direct multimodal embeddings pass images and text to a multimodal embedding model that produces vectors in the same space, need no LLM at indexing time, and suit visual-similarity searches. Image verbalization calls an LLM for every image, which is the cost the team wants to avoid. Semantic ranker reranks search results with language models, and the knowledge store only holds extracted images for client apps to return.
Deep dive
How it works
Azure AI Search multimodal search offers two ways to make images searchable, with a clear trade-off:
- Image verbalization followed by text embeddings: the GenAI Prompt skill invokes an LLM during ingestion to write a short natural-language description of each image, which is then embedded as text. You get explainable, rich semantics, but it costs an LLM call for every image.
- Direct multimodal embeddings: images and text go to a multimodal embedding model that produces vectors in the same vector space. Configuration is straightforward and no LLM is required at indexing time.
The page says direct embeddings are well suited to visual similarity and "find-me-something-that-looks-like-this" scenarios. That's this retailer. At query time, the shopper's photo is embedded with the same model and a vector search returns the nearest product images.
The trade-off: because the representation is purely mathematical, it doesn't convey why two images are related. For "find similar-looking items", that's fine.
The other options, and when you'd use them
- Semantic ranker on descriptions: Semantic ranker uses language models to rerank text search results. It doesn't compare a photo with images.
- Image verbalization with GenAI Prompt: It invokes an LLM for each image during ingestion, which is the cost the team wants to avoid.
- Store photos in the knowledge store: The knowledge store holds extracted images that can be returned to client apps. It's storage, not similarity search.
Exam tip
"Looks like this photo" + no LLM at indexing = direct multimodal embeddings. Need explainable descriptions = verbalization (LLM per image).
References
Last reviewed: