docling-project / docling-project/docling

Address transformers "false-alarm" warning in hybrid chunker

Open
#725 1 comment 1 reaction 1 assignee Claimed by @vagenas View on GitHub
chunker enhancement triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Requested feature

Address the "false-alarm" message from the transformers library emitted during hybrid chunking & discussed [here](https://github.com/DS4SD/docling-core/issues/119).

This could be done by catching the warning using [this](https://docs.python.org/3/library/warnings.html#testing-warnings).
HuggingFace is using a variant where they emit an additional warning, calling to ignore the previous warning ([ref](https://github.com/search?q=repo%3Ahuggingface%2Ftransformers+%22token+indices+sequence+length+is+longer%22&type=code)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.