Implement text analysis solutions 4.1 Apply language model text analysis
A compliance team wants a model to write regulatory summaries in the team's preferred…
Study guide skill: Customize language model outputs for domain tasks, such as compliance summarization and domain extraction
Question B3-026
A compliance team wants a model to write regulatory summaries in the team's preferred style. For each of several thousand source documents, reviewers compared two candidate summaries and marked the one they prefer. Quality depends on tone and completeness rather than one correct answer. Which customization method fits this training data?
Select one answer.
Answer: A
DPO trains on preferred and rejected responses to the same input and fits when quality depends on preferences such as tone or completeness rather than one correct answer. SFT trains on prompts with desired responses that you demonstrate, and RFT needs a grader that scores generated responses. Fine-tuning can align a model's tone, format, and structure, which retrieving example summaries doesn't train into the model.
Deep dive
How it works
The fine-tuning overview picks the method by training signal, the kind of data you have:
- Supervised fine-tuning (SFT): prompts and desired responses. It fits when you can demonstrate the behavior the model should learn.
- Preference fine-tuning with direct preference optimization (DPO): preferred and rejected responses to the same input. It fits when quality depends on preferences such as tone or completeness, rather than one correct answer.
- Reinforcement fine-tuning (RFT): prompts and a grader that scores generated responses. It fits when multiple solutions are possible and you can reliably score their quality.
The compliance team's data is pairs of candidate summaries for the same document with one marked as preferred. That's DPO's training signal word for word, and "tone and completeness rather than one correct answer" is its "when it fits" column.
Fine-tuning is the right family because the goal is style. The overview lists aligning style and structure among its benefits: responses can follow a consistent tone, format, or schema. Benefits aren't guaranteed, so compare the tuned model with your baseline on held-out evaluation data.
The other options, and when you'd use them
- Supervised fine-tuning (SFT): SFT needs prompts with desired responses that demonstrate the behavior. You could train on just the winning summaries, but you'd throw away the preference signal the reviewers created.
- Reinforcement fine-tuning (RFT): RFT needs a grader that scores generated responses. The team has human pairwise choices, not a reliable automatic scorer.
- Index the preferred summaries and use RAG: Retrieval supplies information at query time. The overview notes that fine-tuning doesn't replace retrieval for current information, and the reverse holds: retrieval doesn't train a consistent style into the model.
Exam tip
Data type picks the method: demonstrations = SFT; preferred vs rejected pairs = DPO; grader scores = RFT.
References
Last reviewed: