docling-project / docling-project/docling

Use flash attention with VLM

Open
#1,730 9 comments 3 reactions 0 assignees View on GitHub
question triage/close-fixed
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Question

I am trying to execute https://docling-project.github.io/docling/examples/compare_vlm_models/ with flash attention enabled:

```python
pipeline_options.accelerator_options.cuda_use_flash_attention2 = True
```

This fails with:

```

FlashAttention only support fp16 and bf16 data type You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour docling see when enabling

```

What do I need to do to properly enable flash attention?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.