docling-project / docling-project/docling
Update de semchunk for the chunking methods
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Requested feature
While trying to determine an appropriate chunk size, I noticed that the current solution does not include a basic token overlap feature. This feature allows for a more contextualized partitioning of data in cases where a whole section cannot be fully included.
I understand that some users may not be in favor of overlapping elements because they can negatively impact data retrieval from a vector database. Depending on the overlap size and the retrieval technique used, this could lead to multiple instances of the same content being fetched.
### Alternatives
I have observed that the latest version of **semchunk** already includes overlap support and other useful features, in addition to improving the chunking mechanism.
Contributor guide
Assessment
This issue has not been assessed yet.