crewAIInc / crewAIInc/crewAI

[BUG] RAG text splitter repeats zero-overlap content and exceeds chunk_size

Open
#7,470 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

RecursiveCharacterTextSplitter repeats source text with chunk_overlap=0 and can exceed its configured maximum chunk size. Retained separators are also inserted twice during merging.

Reproduction
from crewai_tools.rag.chunkers.base_chunker import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=3, chunk_overlap=0, separators=[""])
print(splitter.split_text("abcdef"))
# Actual: ["abc", "cde", "ef"]
# Expected: ["abc", "def"]

splitter = RecursiveCharacterTextSplitter(chunk_size=5, chunk_overlap=0, separators=["\n", ""])
print(splitter.split_text("aa\nbb\ncc"))
# Actual: ["aa\n\nbb", "\nbb\n\ncc"] (lengths 6 and 7)
# Expected: ["aa\nbb", "\ncc"] (lengths 5 and 3)
Expected behavior

Merging respects both chunk_size and chunk_overlap, without adding separators that are already present. The excess content otherwise changes retrieval chunks and embedding input sizes.

Cause and verification

The overlap loop always retains one split, even for zero overlap, and does not make room for the next split. Separator accounting assumes removed separators even when they were retained. Four regressions fail on current main; a local fix passes all 137 RAG tests, including overlap and separator controls. Python 3.12 on macOS arm64; no LLM calls.

AI assistance was used to investigate, implement and test this change.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with RecursiveCharacterTextSplitter in crewai_tools/rag/chunkers/base_chunker.py and inspect its split_text merging behavior for zero overlap and retained separators. Reproduce both examples, then run the RAG test suite; done means all 137 RAG tests pass while chunks respect chunk_size and chunk_overlap without duplicated separators.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.