Implement information extraction solutions 5.2 Extract content from documents
A Document Intelligence custom template model trained on one supplier's purchase orders…
Study guide skill: Extract information by using multimodal pipelines that combine OCR, layout analysis, and field extraction
Question B5-028
A Document Intelligence custom template model trained on one supplier's purchase orders reported a high estimated accuracy. After the supplier redesigned its form, documents in the new layout return low field confidence, while the OCR word confidence stays high. What should you do?
Select one answer.
Answer: D
High accuracy with low confidence indicates that the analyzed documents differ from the training data, so add labeled documents with the new format and retrain, or train a model for the new variation and compose the models into one. Read (OCR) confidence doesn't affect key-value confidence, so improving input quality isn't the fix when the OCR confidence is already high. Confidence is meant for deciding whether to auto-accept or send results to human review, and accuracy scores can be inconsistent when documents differ from the training set.
Deep dive
How it works
Document Intelligence custom models report two different numbers, and reading them together tells you what's wrong:
- Accuracy (estimated at training) predicts how well the model handles documents like its training set.
- Confidence (at analysis time) measures how sure the model is about each extracted field on this document.
The accuracy-and-confidence guidance covers this exact combination. High accuracy with low confidence means the analyzed document looks different from the training dataset, a format variation. The fixes:
- Retrain with new samples: label documents in the new layout (the guidance suggests at least five more) and retrain the existing model.
- Or add a model: if the layouts are visually distinct, train a separate model for each variation and compose them into a single model.
The general rule: if the documents look similar once you remove the entered values, add training data; if they're dissimilar, split into separate models.
Word (OCR) confidence is a separate signal. A high read confidence just shows the scan is fine, and read confidence doesn't affect key-value confidence.
The other options, and when you'd use them
- High training accuracy guarantees results: Reported accuracy can be inconsistent when analyzed documents differ from the training documents, which is what happened.
- Lower the auto-accept threshold: Confidence exists to decide whether to accept a prediction automatically or flag it for review. Lowering the threshold only lets wrong values through.
- Improve scan quality: That's the fix for low read confidence. Here OCR confidence is high, and read confidence doesn't affect key-value confidence.
Exam tip
Document Intelligence: high accuracy + low confidence = new layout; label and retrain, or train a model for it and compose. Low OCR confidence = fix input quality.
References
Last reviewed: