microsoft / microsoft/onnxruntime

Implement bias addition during softmax computation in GQA CPU operator

Open
#23,982 0 comments 0 reactions 1 assignee Claimed by @derdeljan-msft View on GitHub
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

In https://github.com/microsoft/onnxruntime/pull/23944, attention bias is applied as a separate pass before softmax computation in GQA and adds attention bias values to the logits. Optimal solution would be to add the attention bias during reduce computation in softmax.

cc @fajin-corp

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.