[AutoDeploy] Expose logit_softcap parameter in flashinfer_attention
Open
@nvchenghaoz is already working on this.
Since Jun 3, 2025.
AutoDeploy
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Flashinfer supports logit_softcap in attention. However, we currently do not expose this parameter in the flashinfer attention interface. We should expose so we can later set it as needed.
More detailed description:
- Just like scale is a constant in the FlashInfer Attention Backend in tensorrt_llm/_torch/auto_deploy/custom_ops/flashinfer_attention, we also want logit_softcap from flashinfer to be a layer-specific constant we can pass through the attention interface.
- Let's expose it but let's set the default to whatever the default is in the flashinfer op so it becomes optional
- Let's also modify the corresponding unit test in the folder tests/unittest/_torch/auto_deploy to test for this new use case / parameter
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.