Implement information extraction solutions 5.1 Build retrieval and grounding pipelines
A RAG agent keeps citing policy PDFs that were deleted from Azure Blob Storage weeks ago.
Study guide skill: Ingest and index content, such as documents, images, audio, and video
Question B3-029
A RAG agent keeps citing policy PDFs that were deleted from Azure Blob Storage weeks ago. The blob indexer runs hourly and picks up new and changed files correctly, but no deletion detection policy was ever configured. What should you do so that the deleted PDFs disappear from the index and future deletions are handled?
Select one answer.
Answer: C
Indexers detect new and changed blobs through timestamps, but they don't track deletions, so orphaned search documents remain unless you use a soft delete strategy. For native blob soft delete, enable soft delete for blobs (blob versioning must not be enabled) and set a native blob soft delete detection policy on the data source. The policy must apply from the very first indexer run: documents deleted before it existed stay in the index even if you add the policy later and reset the indexer, so the guidance is to create a new index with a new indexer that has the policy from the beginning. Versioning prevents native soft delete, and doing nothing leaves the orphans in place.
Deep dive
How it works
For Azure Storage, indexers detect new and changed content automatically by using the storage timestamps. Deletions are different: an indexer doesn't track object deletion. To avoid orphan search documents, you use a soft delete strategy, where the search document is removed first and the physical deletion in storage follows. There are two ways to do it: native blob soft delete (Blob Storage only) or soft delete using custom metadata.
Native blob soft delete has requirements:
- Enable soft delete for blobs on the storage account, with a retention period much longer than your indexer schedule.
- Blob versioning must not be enabled. If it is, native soft delete isn't supported, by design.
- Map document keys to a blob property or metadata, such as metadata_storage_path.
- Set a native blob soft deletion detection policy on the data source (the Track deletions checkbox in the portal, or the REST API).
The catch is timing. The deletion policy must apply from the very first indexer run. Documents deleted before the policy existed stay in the index even if you add the policy later and reset the indexer. For that case, the docs suggest creating a new index with a new indexer that has the deletion policy from the start. That's why both parts of the answer are needed: the fresh index clears the existing orphans, and the policy handles future deletions.
The other options, and when you'd use them
- Enable blob versioning: This makes things worse. Native soft delete isn't supported when versioning is enabled.
- Add the policy to the existing data source and reset: That handles future deletions, but the docs say documents deleted before the policy existed remain in the index even after a reset. The old PDFs would keep showing up.
- Do nothing: Indexers don't track deletions, so the orphaned documents stay indefinitely.
Exam tip
"Deleted blobs still in the index" means soft delete detection. If the policy wasn't there from the first run, rebuild with a new index and indexer. Keep blob versioning off for native soft delete.
References
Last reviewed: