microsoft / microsoft/onnxruntime
Implement bias addition during softmax computation in GQA CPU operator
Open
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
In https://github.com/microsoft/onnxruntime/pull/23944, attention bias is applied as a separate pass before softmax computation in GQA and adds attention bias values to the logits. Optimal solution would be to add the attention bias during reduce computation in softmax.
cc @fajin-corp
Contributor guide
Assessment
This issue has not been assessed yet.