huggingface / huggingface/candle

Retrieving softmax_lse in candle-flash-attn

Open
#2,290 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hello,

In flash-attn, the logsumexp of the softmax is not output. But it would be nice if it could be output too as it is necessary to compute long context KV cache (with ring attention for example). The option to retrieve softmax_lse is present in the python interface of the original flash-attention repo.
https://github.com/Dao-AILab/flash-attention/blob/184b992dcb2a0890adaa19eb9b541c3e4f9d2a08/flash_attn/flash_attn_interface.py#L482C4-L482C4

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing candle-flash-attn's implementation with the referenced flash_attn_interface.py entry point, which exposes softmax_lse. Trace how attention outputs are represented in Candle and identify the relevant tests or examples; the work is done when callers can retrieve softmax_lse alongside the existing attention output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.