[BUG] RAG text splitter repeats zero-overlap content and exceeds chunk_size
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 58.8k
- Forks
- 8.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 109
Description
Description
RecursiveCharacterTextSplitter repeats source text with chunk_overlap=0 and can exceed its configured maximum chunk size. Retained separators are also inserted twice during merging.
Reproduction
from crewai_tools.rag.chunkers.base_chunker import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(chunk_size=3, chunk_overlap=0, separators=[""])
print(splitter.split_text("abcdef"))
# Actual: ["abc", "cde", "ef"]
# Expected: ["abc", "def"]
splitter = RecursiveCharacterTextSplitter(chunk_size=5, chunk_overlap=0, separators=["\n", ""])
print(splitter.split_text("aa\nbb\ncc"))
# Actual: ["aa\n\nbb", "\nbb\n\ncc"] (lengths 6 and 7)
# Expected: ["aa\nbb", "\ncc"] (lengths 5 and 3)
Expected behavior
Merging respects both chunk_size and chunk_overlap, without adding separators that are already present. The excess content otherwise changes retrieval chunks and embedding input sizes.
Cause and verification
The overlap loop always retains one split, even for zero overlap, and does not make room for the next split. Separator accounting assumes removed separators even when they were retained. Four regressions fail on current main; a local fix passes all 137 RAG tests, including overlap and separator controls. Python 3.12 on macOS arm64; no LLM calls.
AI assistance was used to investigate, implement and test this change.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with RecursiveCharacterTextSplitter in crewai_tools/rag/chunkers/base_chunker.py and inspect its split_text merging behavior for zero overlap and retained separators. Reproduce both examples, then run the RAG test suite; done means all 137 RAG tests pass while chunks respect chunk_size and chunk_overlap without duplicated separators.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100