microsoft / microsoft/azurechat
Unchecked growth in Search Service due to documents not being deleted when CosmosDB TTL expires
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1.4k
- Forks
- 1.3k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 1
Description
Description
We've encountered an issue within the azurechat application related to document indexing and deletion. Although the application is designed to upload documents to Azure Search Service, it seems that deletion of documents is not handled properly in all scenarios.
Problem
When users upload documents to chat in azurechat, these documents are indexed in Azure Search Service. When chat threads are deleted via the UI, the deleteDocuments function within the azure-cog-vector-store.ts correctly removes the corresponding documents from the Search Service index.
However, if the thread is automatically removed once the time-to-live (TTL) period for the Cosmos DB history table is reached, the associated documents remain in the search index. This is because the deleteDocuments function is not triggered when the TTL mechanism deletes the thread.
Consequence
This leads to unchecked growth in the Search Service as orphaned documents accumulate, which becomes problematic over time. This incurs unnecessary costs, given that the Search Service is a premium component of this application.
Potential Solutions
Implement a background process that checks for documents associated with threads that are close to their TTL and explicitly calls deleteDocuments before TTL expiration.
Reevaluate the TTL strategy by possibly introducing a soft-delete mechanism where records are flagged as deleted and then purged systematically by a cleanup service.
Request
We need to design and implement a solution that ensures documents are consistently deleted from the Search Service index when their corresponding threads are no longer present. It's critical for managing expenses associated with the Search Service.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading azure-cog-vector-store.ts and tracing deleteDocuments for UI-deleted chat threads. Then inspect how Cosmos DB TTL removes history and identify an appropriate cleanup entry point from the issue's proposed approaches. Done means documents for expired or deleted threads are consistently removed from Azure Search without unchecked index growth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, typescript
- Domain
- cloud, databases, search
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100