deepset-ai / deepset-ai/haystack

DocumentSplitter emits redundant overlap-only trailing chunks for character-based split modes (token mode was fixed in #12661)

Open
#12,686 0 comments 0 reactions 1 assignee View on GitHub

@davidsbatista is already working on this.

Since Sep 18, 2026.

Dominant language
Python
Stars
26.6k
Forks
3.2k
Avg merge
1d 3h
Merged PRs (30d)
194

Description

Describe the bug
DocumentSplitter with split_by="token" no longer emits overlap-only trailing chunks since #12661, but the character-based modes sharing _concatenate_units (word, period, page, passage, line, sentence) still do. A trailing window consisting solely of already-covered overlap units (plus the empty artifact of a trailing delimiter) becomes its own chunk, duplicating text that is already fully contained in the previous chunk.

To Reproduce

from haystack import Document
from haystack.components.preprocessors import DocumentSplitter

splitter = DocumentSplitter(split_by="period", split_length=3, split_overlap=1)
docs = splitter.run(documents=[Document(content="s1.s2.s3.")])["documents"]
print([d.content for d in docs])
# actual:   ['s1.s2.s3.', 's3.']  <- 's3.' adds no new text
# expected: ['s1.s2.s3.']

High overlap shows it without any trailing delimiter:

splitter = DocumentSplitter(split_by="period", split_length=3, split_overlap=2)
docs = splitter.run(documents=[Document(content="s1.s2.s3.s4.")])["documents"]
print([d.content for d in docs])
# actual:   ['s1.s2.s3.', 's2.s3.s4.', 's3.s4.']  <- third chunk is fully contained in the second
# expected: ['s1.s2.s3.', 's2.s3.s4.']

Expected behavior
Mirror the split_by="token" behavior from #12661: never emit a chunk that adds no new text. Partial final chunks carrying new content (e.g. word mode "t1 t2 t3 t4" with length 3 / overlap 1 -> ['t1 t2 t3 ', 't3 t4']) and below-threshold merges must keep working.

Additional context
Root cause: _concatenate_units in haystack/components/preprocessors/document_splitter.py emits every trailing window yielded by more_itertools.windowed as long as its text is non-empty, without checking whether the window reaches beyond the already covered units.

Possible fixes:

  • (a) skip non-first windows whose units past the covered boundary are all empty (mirrors the token-mode fix; threshold merging of genuine partial tails is preserved);
  • (b) only strip the trailing empty delimiter artifact (fixes exact-fit inputs but leaves high-overlap tails like the second repro);
  • (c) leave as-is (rejected: duplicate chunks pollute document stores, and I found one such phantom chunk even carrying a wrong page_number).

I have a fix along the lines of (a) with regression tests ready and can open a PR.

FAQ Check

System:

  • OS: macOS (arm64)
  • Haystack version: main @ b20727a (post-#12661)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.