Chunking Testbed
Open
@ddrechse is already working on this.
Since Apr 17, 2025.
- Dominant language
- Python
- Stars
- 101
- Forks
- 46
- Avg merge
- 6h 19m
- Merged PRs (30d)
- 23
Description
Look into having a “Chunking Testbed” or some way to generate a single report for evaluating the LLM generated questions via a vector store where chunks were created using different chunking strategies? Also, there are several embedding models that have built-in chunking – e.g. jina ai has late-chunking which was proven to be better.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.