Implement computer vision solutions 3.2 Design and implement multimodal understanding workflows
An app asks a vision-enabled chat model questions about single photos of equipment…
Study guide skill: Implement a solution that enables question-answering grounded in visual evidence
Question B3-020
An app asks a vision-enabled chat model questions about single photos of equipment nameplates. Answers are often vague, and the model sometimes refuses. Which two prompt changes does Microsoft's image prompt guidance recommend? (Choose two.)
Select 2 answers.
Answer: D, E
For refusals and weak answers, the guidance is to refine the prompt, place the image before the text for single-image prompts, and ask the model to describe the image in detail before completing the task. Adding scenario context helps the model focus on relevant details, so removing it works against you. The guidance also recommends stating the desired output format clearly.
Deep dive
How it works
Vague answers and refusals on image questions usually mean the prompt isn't steering the model well enough. Microsoft's prompt guidance for vision-enabled chat models has a specific playbook for this, under Handle refusals: when the model indicates it can't do a task, refine the prompt. More specific prompts guide the model toward a clearer understanding of the task.
The two concrete changes in the correct answers come from that same guidance:
- Place the image before the text for single-image prompts.
- Ask the model to describe the image in detail first, then complete your specific task. Describing the nameplate first makes the model attend to the text and markings before answering.
Other techniques on the page that fit a nameplate app:
- Contextual specificity: add context about the scenario so the model focuses on relevant details and avoids extraneous ones.
- Task-oriented prompts: focus on the specific task, such as reading the model number and rated voltage.
- Define output format: clearly state the format you want, such as JSON with named fields.
The other options, and when you'd use them
- Place the image after long text instructions: The guidance is the reverse: place the image before the text for single-image prompts.
- Avoid specifying an output format: The guidance says to clearly mention the desired output format, such as markdown, JSON or HTML, and you can suggest a structure or length.
- Remove all scenario context: Context helps. Adding context to the scenario gives the model a better understanding of an appropriate output and helps it focus on relevant aspects.
Exam tip
Vision prompt fixes: image first, "describe the image in detail, then answer", add scenario context, and state the output format.
References
Last reviewed: