docling-project / docling-project/docling
Stride - chunk overlap
Open
chunker
enhancement
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 98
Description
### Requested feature
Would be great to have a param that can go into the chunker to enforce chunk overlap.
something like this:
# Docling chunking initialisation
openai_tokenizer = OpenAITokenizer(
tokenizer=tiktoken.encoding_for_model(model_embedding),
max_tokens=500, # Token limit
stride=50, # Token overlap
)
Contributor guide
Assessment
This issue has not been assessed yet.