docling-project / docling-project/docling

Update de semchunk for the chunking methods

Open
#1,281 1 comment 10 reactions 0 assignees View on GitHub
enhancement triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature

While trying to determine an appropriate chunk size, I noticed that the current solution does not include a basic token overlap feature. This feature allows for a more contextualized partitioning of data in cases where a whole section cannot be fully included.

I understand that some users may not be in favor of overlapping elements because they can negatively impact data retrieval from a vector database. Depending on the overlap size and the retrieval technique used, this could lead to multiple instances of the same content being fetched.

### Alternatives

I have observed that the latest version of **semchunk** already includes overlap support and other useful features, in addition to improving the chunking mechanism.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.