docling-project / docling-project/docling

[datatype] how to handle bfloat16

Open
#3,110 1 comment 0 reactions 0 assignees View on GitHub
question triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

Currently, docling loads modesl with their default values from transformers.

It also allows to override dtypes (in some places). One of these is the picture classifier.

Setting
```
picture_engine_options.torch_dtype = "bfloat16"
```

fails with

```
RuntimeError: mat1 and mat2 must have the same dtype, but got Float and BFloat16
```

In particular I was facing this with https://github.com/mindee/doctr however, this seems to be a more general pattern that docling is not focusing on too much for now? Or is it the intention to use the remote VLM (via perhaps VLLM/ ...) and then onfigure the exact settings there?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.