docling-project / docling-project/docling
Use flash attention with VLM
Open
question
triage/close-fixed
- Dominant language
- Python
- Stars
- 66.4k
- Forks
- 4.8k
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 84
Description
### Question
I am trying to execute https://docling-project.github.io/docling/examples/compare_vlm_models/ with flash attention enabled:
```python
pipeline_options.accelerator_options.cuda_use_flash_attention2 = True
```
This fails with:
```
FlashAttention only support fp16 and bf16 data type You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour docling see when enabling
```
What do I need to do to properly enable flash attention?
Contributor guide
Assessment
This issue has not been assessed yet.