Implement computer vision solutions 3.2 Design and implement multimodal understanding workflows
An insurance app passes image URLs to a vision-enabled chat model.
Study guide skill: Build a solution that analyzes visual context by using multimodal models
Question B5-004
An insurance app passes image URLs to a vision-enabled chat model. The claim photos are in a storage account that can be reached only through a private endpoint in the company's virtual network, and the model can't process them. What should you do?
Select one answer.
Answer: A
The vision service fetches image URLs from Microsoft's managed infrastructure, not from inside your virtual network, so private-endpoint, VNet-restricted, and firewall-restricted URLs aren't supported, even on a peered network. Sending the image as base64-encoded data instead of a URL avoids the fetch entirely. The SAS and managed identity setup applies to images the service can reach, and the detail setting only changes how the model processes an image it already has.
Deep dive
How it works
When you pass an image URL to a vision-enabled chat model, the model's service has to fetch that image. The docs are explicit about where the fetch comes from: the vision service fetches images from Microsoft's managed infrastructure, not from within your virtual network. So a request can't see a private endpoint in your VNet. Image URLs must be publicly accessible. Private-endpoint, VNet-restricted and firewall-restricted URLs aren't supported, even if the storage account is in the same or a peered VNet.
The way around it is to stop asking the service to fetch anything. Message content can be an array of text and images, where each image is either a publicly reachable HTTP or HTTPS URL or base64-encoded image data. Your app, which can reach the private storage, reads the file and sends it inline as a data URL in the image_url content part:
- Read the image bytes in the app.
- Base64-encode them and build a data URL such as data:image/jpeg;base64,...
- Put that data URL in the image_url field of the message.
The docs include a small helper that guesses the MIME type and builds the data URL for you. Other request limits still apply, such as up to 10 images per chat request and the JPEG, PNG, GIF (first frame) and WEBP formats.
The other options, and when you'd use them
- Set detail to low: The detail setting (low, high or auto) changes how the model processes an image it already has. It doesn't change how, or from where, the image is fetched.
- SAS URLs plus the Storage Blob Reader role for the resource's managed identity: The docs describe this setup for images in Blob Storage that the service can access. It doesn't get around the private endpoint, because the fetch still comes from outside the VNet.
- Move the images to a storage account in a peered VNet: The docs specifically say peered VNets aren't supported either.
Exam tip
"Images behind a private endpoint or firewall" means you send base64 data, not URLs. The service fetches URLs from Microsoft's infrastructure, outside your network.
References
Last reviewed: