Azure-Samples / Azure-Samples/chat-with-your-data-solution-accelerator
Implement more efficient chunking strategies
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 648
- Avg merge
- 1d 30m
- Merged PRs (30d)
- 9
Description
# Motivation
Currently the chunking is possible using Layout, Page, Fixed-size and Paragraph strategies with possible overlap. I would suggest an additional strategy focused solely on quality: LLM chunking. A LLM is called to have coherent and relevant chunks, each chunk expressing the same idea or concept or thought.
In all chunking strategies, a LLM can be used to generate additional metadata to improve the reranking by Azure AI Search when semantic search is True. The LLM would populate for each chunk:
- "Title" by summarizing the chunk into one short sentence
- "Keyword" by extracting the main keywords of the chunk
Please note that in order to make the reranking useful, [ticket 2093](https://github.com/Azure-Samples/chat-with-your-data-solution-accelerator/issues/2093) needs to be implemented.
# Tasks
_To be filled in by the engineer picking up the issue_
- [ ] Task 1
- [ ] Task 2
- [ ] ...
Contributor guide
Research direction
Start by reviewing the existing Layout, Page, Fixed-size, and Paragraph chunking strategies and the Azure AI Search semantic-search path. Define the LLM chunking and per-chunk Title/Keyword metadata requirements, then check how ticket 2093 affects reranking. Done should include concrete tasks and an agreed implementation scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- ai, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100