huggingface / huggingface/candle
Retrieving softmax_lse in candle-flash-attn
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Hello,
In flash-attn, the logsumexp of the softmax is not output. But it would be nice if it could be output too as it is necessary to compute long context KV cache (with ring attention for example). The option to retrieve softmax_lse is present in the python interface of the original flash-attention repo.
https://github.com/Dao-AILab/flash-attention/blob/184b992dcb2a0890adaa19eb9b541c3e4f9d2a08/flash_attn/flash_attn_interface.py#L482C4-L482C4
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing candle-flash-attn's implementation with the referenced flash_attn_interface.py entry point, which exposes softmax_lse. Trace how attention outputs are represented in Candle and identify the relevant tests or examples; the work is done when callers can retrieve softmax_lse alongside the existing attention output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, rust
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100