Improve our embedding splitting
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 970
- Forks
- 486
- Avg merge
- 3d 33m
- Merged PRs (30d)
- 170
Description
Description
It seems it's still best practices to have a roughly 500 token count split for the embeddings. And we are also splitting on sentences, which has been good. But what we haven't done is had overlap. And generally speaking, it's good to have 15 to 20% overlap between splits so that you maintain context. This does that.
Acceptance Criteria
- Only affects future embeddings
- existing Embeddings continue to work
Priority
None
Additional Context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file or test is named. Start by locating the Java implementation that splits text for embeddings and checking how existing embeddings are stored; then trace the embedding-generation entry point. Done means future embeddings use roughly 500-token sentence-based splits with 15–20% overlap, while existing embeddings remain unchanged, with tests covering both behaviors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100