deepset-ai / deepset-ai/haystack
DocumentSplitter emits redundant overlap-only trailing chunks for character-based split modes (token mode was fixed in #12661)
@davidsbatista is already working on this.
Since Sep 18, 2026.
- Dominant language
- Python
- Stars
- 26.6k
- Forks
- 3.2k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 194
Description
Describe the bug
DocumentSplitter with split_by="token" no longer emits overlap-only trailing chunks since #12661, but the character-based modes sharing _concatenate_units (word, period, page, passage, line, sentence) still do. A trailing window consisting solely of already-covered overlap units (plus the empty artifact of a trailing delimiter) becomes its own chunk, duplicating text that is already fully contained in the previous chunk.
To Reproduce
from haystack import Document
from haystack.components.preprocessors import DocumentSplitter
splitter = DocumentSplitter(split_by="period", split_length=3, split_overlap=1)
docs = splitter.run(documents=[Document(content="s1.s2.s3.")])["documents"]
print([d.content for d in docs])
# actual: ['s1.s2.s3.', 's3.'] <- 's3.' adds no new text
# expected: ['s1.s2.s3.']
High overlap shows it without any trailing delimiter:
splitter = DocumentSplitter(split_by="period", split_length=3, split_overlap=2)
docs = splitter.run(documents=[Document(content="s1.s2.s3.s4.")])["documents"]
print([d.content for d in docs])
# actual: ['s1.s2.s3.', 's2.s3.s4.', 's3.s4.'] <- third chunk is fully contained in the second
# expected: ['s1.s2.s3.', 's2.s3.s4.']
Expected behavior
Mirror the split_by="token" behavior from #12661: never emit a chunk that adds no new text. Partial final chunks carrying new content (e.g. word mode "t1 t2 t3 t4" with length 3 / overlap 1 -> ['t1 t2 t3 ', 't3 t4']) and below-threshold merges must keep working.
Additional context
Root cause: _concatenate_units in haystack/components/preprocessors/document_splitter.py emits every trailing window yielded by more_itertools.windowed as long as its text is non-empty, without checking whether the window reaches beyond the already covered units.
Possible fixes:
- (a) skip non-first windows whose units past the covered boundary are all empty (mirrors the token-mode fix; threshold merging of genuine partial tails is preserved);
- (b) only strip the trailing empty delimiter artifact (fixes exact-fit inputs but leaves high-overlap tails like the second repro);
- (c) leave as-is (rejected: duplicate chunks pollute document stores, and I found one such phantom chunk even carrying a wrong
page_number).
I have a fix along the lines of (a) with regression tests ready and can open a PR.
FAQ Check
- Have you had a look at our new FAQ page?
System:
- OS: macOS (arm64)
- Haystack version: main @ b20727a (post-#12661)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.