docling-project / docling-project/docling
[datatype] how to handle bfloat16
Open
question
triage/close-stale
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
Currently, docling loads modesl with their default values from transformers.
It also allows to override dtypes (in some places). One of these is the picture classifier.
Setting
```
picture_engine_options.torch_dtype = "bfloat16"
```
fails with
```
RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
```
In particular I was facing this with https://github.com/mindee/doctr however, this seems to be a more general pattern that docling is not focusing on too much for now? Or is it the intention to use the remote VLM (via perhaps VLLM/ ...) and then onfigure the exact settings there?
Contributor guide
Assessment
This issue has not been assessed yet.