Lightning-AI / Lightning-AI/litgpt

`mask_cache` in kv-cache path seems to force `attn_mask`, preventing flash attention

Open
#2,220 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Python
Stars
13.7k
Forks
1.5k
Avg merge
15h 37m
Merged PRs (30d)
1

Description

Body:

Hi, thank you for this great implementation. I found a behavior that I am confused about and would like clarification:

1. In model.py, `GPT.set_kv_cache()` always does:
```python
if self.mask_cache is None or self.mask_cache.size(3) != max_seq_length:
# passing `attn_mask` to SDPA disables the flash implementation...
self.mask_cache = build_mask_cache(max_seq_length, device)
```
2. In `GPT.forward()`, if `input_pos is not None`:
```python
if self.mask_cache is None:
raise TypeError("You need to call `gpt.set_kv_cache()`")
mask = batched_index_select(self.mask_cache, 2, input_pos)
```
3. In `CausalSelfAttention.scaled_dot_product_attention()`:
```python
y = F.scaled_dot_product_attention(
q, k, v, attn_mask=mask, ..., is_causal=mask is None
)
```
So it looks like kv-cache path always uses `attn_mask` (non-null), which would set `is_causal=False` in PyTorch SDPA and appear to avoid flash attention path.
- Is this intentional design?
- Is there any recommended usage mode where kv-cache and flash-attn fast path are both active?
- Could this be a misunderstanding on my side?

Thanks for any clarification.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in model.py by tracing GPT.set_kv_cache(), GPT.forward(), and CausalSelfAttention.scaled_dot_product_attention(). Check how the selected mask reaches PyTorch SDPA and whether the kv-cache path permits the flash attention implementation. Done means documenting the intended behavior or identifying a concrete change needed to use both kv-cache and flash attention.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.