memory_required() estimate misses masked attention fallback and per-model attention overrides
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
**What we found**
Two gaps in the flash-attention memory estimate added by #15586, both flagged by Codex on that PR:
1. Masked attention falls back to plain SDPA in `attention_flash()`, not real flash attention. `memory_required()` still returns the cheap estimate for it.
2. The `ModelAttentionBackend` node can override a model's attention backend at runtime. `memory_required()` can't see that override, so it can return the cheap estimate on a model that isn't actually running flash attention.
**What we want**
`memory_required()` should never return the cheap estimate when the real attention path isn't flash.
Follow-up to #15586, tracking the deferred items from that review.
Contributor guide
Research direction
Start by locating memory_required(), attention_flash(), and the ModelAttentionBackend node, then inspect the flash-attention change from #15586. Trace how masked attention and runtime backend overrides affect the selected path. Done means memory_required() no longer returns the cheap estimate unless the actual attention path is flash, with coverage for both reported gaps.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100