Lightning-AI / Lightning-AI/litgpt
`mask_cache` in kv-cache path seems to force `attn_mask`, preventing flash attention
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
Body:
Hi, thank you for this great implementation. I found a behavior that I am confused about and would like clarification:
1. In model.py, `GPT.set_kv_cache()` always does:
```python
if self.mask_cache is None or self.mask_cache.size(3) != max_seq_length:
# passing `attn_mask` to SDPA disables the flash implementation...
self.mask_cache = build_mask_cache(max_seq_length, device)
```
2. In `GPT.forward()`, if `input_pos is not None`:
```python
if self.mask_cache is None:
raise TypeError("You need to call `gpt.set_kv_cache()`")
mask = batched_index_select(self.mask_cache, 2, input_pos)
```
3. In `CausalSelfAttention.scaled_dot_product_attention()`:
```python
y = F.scaled_dot_product_attention(
q, k, v, attn_mask=mask, ..., is_causal=mask is None
)
```
So it looks like kv-cache path always uses `attn_mask` (non-null), which would set `is_causal=False` in PyTorch SDPA and appear to avoid flash attention path.
- Is this intentional design?
- Is there any recommended usage mode where kv-cache and flash-attn fast path are both active?
- Could this be a misunderstanding on my side?
Thanks for any clarification.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in model.py by tracing GPT.set_kv_cache(), GPT.forward(), and CausalSelfAttention.scaled_dot_product_attention(). Check how the selected mask reaches PyTorch SDPA and whether the kv-cache path permits the flash attention implementation. Done means documenting the intended behavior or identifying a concrete change needed to use both kv-cache and flash attention.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100