Implement computer vision solutions 3.2 Design and implement multimodal understanding workflows

An app asks a vision-enabled chat model 'What is this dog breed, and what are its average…

Study guide skill: Implement a solution that enables question-answering grounded in visual evidence

Question B7-005

An app asks a vision-enabled chat model 'What is this dog breed, and what are its average weight and height?' about user photos and saves the answer to a database. The answers vary in structure, so parsing them often fails. Which two prompt changes from Microsoft's image prompt guidance help most? (Choose two.)

Select 2 answers.

  • A Remove any context from the prompt so the model isn't biased toward a domain
  • B State the required output format clearly, for example JSON with named attributes ✓ Correct answer
  • C Add examples of the expected answers, such as 'Breed: ...; average weight: ...; average height: ...' ✓ Correct answer
  • D Put the text question before the image in the message
  • E Ask the breed, weight, height and care tips in one complex request instead of separate steps

Answer: B, C

Microsoft's image prompt guidance recommends adding examples that show the type of response you want and clearly stating the output format, such as JSON, with its structure and attributes. For single-image prompts, it says to place the image before the text. It also recommends adding context and breaking complex requests into steps, which rules out the other options.

Deep dive

How it works

Microsoft's guidance for prompting vision-enabled models (GPT-4o, GPT-4o-mini and GPT-4 Turbo with Vision) lists a few basics. Two of them fix the parsing problem in the question:

  • Define output format: Clearly state the format you want, such as markdown, JSON or HTML. You can also specify a structure, length or particular attributes. Asking for JSON with named fields like breed, averageWeight and averageHeight gives the database code something predictable to parse.
  • Add examples: Include examples of the kind of response you're looking for. One or two sample answers in the exact shape you want show the model the pattern to follow.

Used together, the format instruction says what the structure is and the examples show it. That's what stops the answers from drifting between prose, lists and sentences.

The same page has more guidance worth knowing for the exam:

  • Contextual specificity: Adding context about the scenario helps the model focus on relevant details and skip extraneous ones.
  • Task-oriented prompts: Giving the model a specific task sharpens the response.
  • Break down requests: Split complex requests into manageable steps.
  • For single-image prompts, place the image before the text, and consider asking the model to describe the image in detail before doing the specific task.

The other options, and when you'd use them

  • Remove any context from the prompt: The guidance says the opposite. Context helps the model focus on what matters.
  • Put the text question before the image: For single-image prompts, the guidance is to place the image before the text.
  • Ask everything in one complex request: The guidance recommends breaking complex requests into steps, not combining more into a single request.

Exam tip

For "inconsistent structure" problems, the two fixes are "define output format" and "add examples." Also remember the single-image rule: image first, then text.

References

Last reviewed:

Ready to test yourself? The quiz asks this and every other AI-103 question, with instant feedback. Practice this in a quiz →

All Computer vision questions · AI-103 study guide

Thanks for your feedback!